Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 15 of 26

Context engineering

0 of 26 complete

Why it matters

Prompt engineering asks how to phrase an instruction. Context engineering asks what else is in the window when the model reads it. As models have become better at following clear instructions, the marginal quality gain has moved from phrasing to composition: whether the right document was retrieved, whether the stale version was excluded, whether the tool description is unambiguous, whether ten turns of history are crowding out the current question.

The problem is structural. A request is assembled by several subsystems — retrieval, memory, tool registry, conversation store — each of which adds content for its own reasons. Nobody owns the total. The result is a context that grows over time, costs more, and gets worse in ways that look like model regressions.

Mental model

Briefing an analyst with a fixed-size folder. You choose which documents go in, in what order, with a one-page summary on top and the raw evidence behind it. A thicker folder is not a better brief; the analyst has to find the relevant page.

The analogy breaks because the analyst reads everything at once rather than searching. Attention is not uniform: content at the beginning and end of a long context tends to be used more reliably than content buried in the middle, and irrelevant passages can actively distract. Composition is therefore about salience, not just coverage.

How it works

  1. 1Define what the task requires: which facts, which rules, which tools, which history.
  2. 2Inventory every source that could contribute, and assign each a trust level and an owner.
  3. 3Select per request using relevance and recency rather than fixed quotas.
  4. 4Compress what must stay but does not fit: extract facts, summarise, drop examples.
  5. 5Order deliberately: stable instructions first, decisive evidence near the generation point.
  6. 6Isolate untrusted content in labelled data sections, away from instruction positions.
  7. 7Measure with ablations and per-section token metrics; remove what does not earn its place.
Composing a context windowInteractive

Context window · illustrative tokens

7,200 input+ 1,000 reserve= 8,200 of 8,000

over by 200

Step 1/6 · Sources arrive

Step 1Sources arrive

Every subsystem adds its own content — retrieval, memory, history — and nobody owns the total. The input is 200 tokens past the space left for the answer.

reserve consumed7,200 tokens

Output reserve

1,000
output reservefixedBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state. Drag the window control to see what the budget does to the composition.

Figure. A context is assembled from parts that each arrive for their own reason. Select, compress, order and isolate until the request fits, and keep the output reserve untouched. Token counts are illustrative.

What to notice. The window is 8,000 tokens with a 1,000-token output reserve — smaller than current flagship windows so the pressure is visible. The figure shows composition and fit, not answer quality, and the ablation step describes a measurement rather than reporting one.

Text version

A context window of 8,000 tokens with a 1,000-token output reserve. The default first step packs eight blocks: system instructions 600, tool definitions 300, worked examples 500, saved preferences 400, conversation history 1,800, refund policy 900, escalation matrix 1,100 and a support FAQ export 1,600. The input is 7,200 tokens, 200 past the available 7,000, and a hatched band over the reserve says so. Later steps drop the FAQ and saved preferences, replace history with a 450-token summary, truncate the escalation matrix from 1,100 to 700, mark the retrieved passages as isolated data, and leave the worked examples out as the next ablation. A window slider runs from 4,000 to 9,000 tokens. Selecting any block shows its token count, kind, role and state.

Selection is the highest-leverage step. Retrieve, filter by permission and freshness, deduplicate near-identical chunks, and cap each source so one connector cannot dominate. A context with five strong passages beats one with twenty mediocre ones, and the token saving is a side effect.

Compression is the fallback, not the default. Extracting the sentences that answer the question preserves exact identifiers; summarising a document introduces another model's errors and loses detail. For conversation history, compaction is often necessary, but it should preserve commitments, identifiers and decisions rather than producing a generic recap.

Ordering is cheap and frequently ignored. Stable instructions at the start can be cached and are read first. The decisive evidence belongs close to where generation starts, not in the middle of a long evidence block. If an instruction must survive a long context, repeating a short version at the end is a pragmatic measure, not a substitute for a shorter context.

Isolation is a security property as well as a quality one. Retrieved documents, web pages and tool output are data. They should be delimited, labelled as data, and never concatenated into the instruction section. The model reads one token stream; the structure you impose is the only separation it has.

Quick facts

Scope
The whole input, not the wording
Core moves
Select, compress, order, isolate
Bound
Token budget
Evidence
Ablation on an eval set
Common failure
Unowned growth
Security boundary
Trust labels and tool scopes

Composition as a function

A context is a function of the request and the available sources:

C=order(select(S,q),  compress(⋅,  B))C = \text{order}\big(\text{select}(S, q),\; \text{compress}(\cdot,\; B)\big)

where SS is the candidate set, qq the query, and BB the token budget. Two systems with the same model and prompt can differ entirely in SS and in the selection policy. That is why copying a prompt rarely reproduces a product's quality.

The measurement that matters is marginal: for each section, what happens to task success when it is removed? Sections that never change the outcome are cost. Sections that change it only when combined with another are the ones a naive ablation will miss, so test combinations for the critical path.

Assembling a context

from dataclasses import dataclass
 
@dataclass
class Part:
    name: str
    text: str
    priority: int          # lower is more important
    trust: str             # "trusted" | "untrusted"
    digest: str            # content hash for dedupe
 
def assemble(parts, budget, count, clip):
    seen, kept, used = set(), [], 0
    for part in sorted(parts, key=lambda p: p.priority):
        if part.digest in seen:
            continue
        seen.add(part.digest)
 
        room = budget - used
        if room <= 0:
            break
        tokens = count(part.text)
        text = part.text if tokens <= room else clip(part.text, room)
 
        kept.append({"name": part.name, "text": text, "trust": part.trust})
        used += count(text)
 
    return kept, {"used": used, "remaining": budget - used}

Log the returned metadata with the request. When an answer is wrong, the first artefact to inspect is the assembled context: which parts were included, at what size, in what order, and what was dropped.

Trade-offs

StrategyGood forCostFails when
Full history in contextShort sessions, continuityLinear token growth, distractionConversations run long or span topics
Summarised historyLong chat, bounded costLossy, extra model callExact commitments or identifiers matter
Retrieval-selected contextLarge corpora, changing knowledgeRetrieval latency and tuningRecall is poor or permissions are complex
Static curated contextStable policy and formatManual maintenanceThe domain changes faster than the curation
Long-context model, everything inOne-off analysis of a large artefactCost, latency, middle neglectRequests repeat at volume
Trade-off mapInteractive

Good for

Short sessions, continuity

Cost

Linear token growth, distraction

Fails when

Conversations run long or span topics

  1. Conversations run long or span topics
  2. Exact commitments or identifiers matter
  3. Recall is poor or permissions are complex
  4. The domain changes faster than the curation
  5. Requests repeat at volume

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full history in context is good for Short sessions, continuity, costs Linear token growth, distraction, and fails when Conversations run long or span topics. Summarised history is good for Long chat, bounded cost, costs Lossy, extra model call, and fails when Exact commitments or identifiers matter. Retrieval-selected context is good for Large corpora, changing knowledge, costs Retrieval latency and tuning, and fails when Recall is poor or permissions are complex. Static curated context is good for Stable policy and format, costs Manual maintenance, and fails when The domain changes faster than the curation. Long-context model, everything in is good for One-off analysis of a large artefact, costs Cost, latency, middle neglect, and fails when Requests repeat at volume

Failure modes

Context poisoning. A wrong fact enters memory or a summary and is re-injected on every turn. Debug by tracing the provenance of each context part and testing whether removing the suspect part changes the answer.

Distraction. Irrelevant but topical passages pull the answer off target. Debug by comparing accuracy at different retrieval depths; if accuracy falls as depth rises, the selector is admitting noise.

Context clash. Two sources disagree and the model averages them. Debug by looking for near-duplicate documents with different dates or versions, and resolve at selection time rather than asking the model to adjudicate.

Stale memory. A stored preference or summary outlives its truth. Debug by timestamping memory entries and testing expiry; memory without a freshness policy becomes a liability.

Failure modes, one at a timeInteractive

Symptom

A wrong fact enters memory or a summary and is re-injected on every turn.

Debug step

Tracing the provenance of each context part and testing whether removing the suspect part changes the answer.

Figure. Each failure with its symptom and the check that would confirm it.

What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.

Text version

Failure modes: Context poisoning: A wrong fact enters memory or a summary and is re-injected on every turn. Debug by tracing the provenance of each context part and testing whether removing the suspect part changes the answer. | Distraction: Irrelevant but topical passages pull the answer off target. Debug by comparing accuracy at different retrieval depths; if accuracy falls as depth rises, the selector is admitting noise. | Context clash: Two sources disagree and the model averages them. Debug by looking for near-duplicate documents with different dates or versions, and resolve at selection time rather than asking the model to adjudicate. | Stale memory: A stored preference or summary outlives its truth. Debug by timestamping memory entries and testing expiry; memory without a freshness policy becomes a liability.

In production

Track tokens per section, not just per request, and alert when a section's share grows without a corresponding quality gain. Cache the stable prefix. Hash the assembled context and store it with the trace so a bad answer is reproducible. Run the eval set against context variants — with and without history, retrieval depth 5 versus 15 — because context changes are behaviour changes. Keep an owner for the composition layer; the failure mode is nobody's job.

Common misconceptions

People think: Context engineering is prompt engineering with a new name.

Actually: Prompt engineering shapes instructions. Context engineering decides what evidence, tools, memory and history are present at all, which is often the larger effect.

People think: More context is safer because the model can ignore what it does not need.

Actually: Models do not reliably ignore irrelevant content. Long inputs cost more, run slower, and can reduce accuracy when the relevant passage is buried.

People think: Compaction is a lossless way to keep a conversation going.

Actually: Summaries drop detail. Commitments, identifiers and decisions must be preserved explicitly, or the compacted context will contradict the original.

Interview angle

A common prompt is "quality dropped after we added memory". The strong answer inspects the assembled context, ablates sections, checks ordering and trust labels, and proposes a budget with an owner. The weak answer rewrites the prompt. See the prompting bank and the system design bank.

Sources