Why it matters
Prompt engineering asks how to phrase an instruction. Context engineering asks what else is in the window when the model reads it. As models have become better at following clear instructions, the marginal quality gain has moved from phrasing to composition: whether the right document was retrieved, whether the stale version was excluded, whether the tool description is unambiguous, whether ten turns of history are crowding out the current question.
The problem is structural. A request is assembled by several subsystems — retrieval, memory, tool registry, conversation store — each of which adds content for its own reasons. Nobody owns the total. The result is a context that grows over time, costs more, and gets worse in ways that look like model regressions.
Mental model
Briefing an analyst with a fixed-size folder. You choose which documents go in, in what order, with a one-page summary on top and the raw evidence behind it. A thicker folder is not a better brief; the analyst has to find the relevant page.
The analogy breaks because the analyst reads everything at once rather than searching. Attention is not uniform: content at the beginning and end of a long context tends to be used more reliably than content buried in the middle, and irrelevant passages can actively distract. Composition is therefore about salience, not just coverage.
How it works
- 1Define what the task requires: which facts, which rules, which tools, which history.
- 2Inventory every source that could contribute, and assign each a trust level and an owner.
- 3Select per request using relevance and recency rather than fixed quotas.
- 4Compress what must stay but does not fit: extract facts, summarise, drop examples.
- 5Order deliberately: stable instructions first, decisive evidence near the generation point.
- 6Isolate untrusted content in labelled data sections, away from instruction positions.
- 7Measure with ablations and per-section token metrics; remove what does not earn its place.
Context window · illustrative tokens
7,200 input+ 1,000 reserve= 8,200 of 8,000
over by 200
Step 1Sources arrive
Every subsystem adds its own content — retrieval, memory, history — and nobody owns the total. The input is 200 tokens past the space left for the answer.
Output reserve
1,000Pick a section to read its token count, its role and its state. Drag the window control to see what the budget does to the composition.
Figure. A context is assembled from parts that each arrive for their own reason. Select, compress, order and isolate until the request fits, and keep the output reserve untouched. Token counts are illustrative.
What to notice. The window is 8,000 tokens with a 1,000-token output reserve — smaller than current flagship windows so the pressure is visible. The figure shows composition and fit, not answer quality, and the ablation step describes a measurement rather than reporting one.
Text version
A context window of 8,000 tokens with a 1,000-token output reserve. The default first step packs eight blocks: system instructions 600, tool definitions 300, worked examples 500, saved preferences 400, conversation history 1,800, refund policy 900, escalation matrix 1,100 and a support FAQ export 1,600. The input is 7,200 tokens, 200 past the available 7,000, and a hatched band over the reserve says so. Later steps drop the FAQ and saved preferences, replace history with a 450-token summary, truncate the escalation matrix from 1,100 to 700, mark the retrieved passages as isolated data, and leave the worked examples out as the next ablation. A window slider runs from 4,000 to 9,000 tokens. Selecting any block shows its token count, kind, role and state.
Selection is the highest-leverage step. Retrieve, filter by permission and freshness, deduplicate near-identical chunks, and cap each source so one connector cannot dominate. A context with five strong passages beats one with twenty mediocre ones, and the token saving is a side effect.
Compression is the fallback, not the default. Extracting the sentences that answer the question preserves exact identifiers; summarising a document introduces another model's errors and loses detail. For conversation history, compaction is often necessary, but it should preserve commitments, identifiers and decisions rather than producing a generic recap.
Ordering is cheap and frequently ignored. Stable instructions at the start can be cached and are read first. The decisive evidence belongs close to where generation starts, not in the middle of a long evidence block. If an instruction must survive a long context, repeating a short version at the end is a pragmatic measure, not a substitute for a shorter context.
Isolation is a security property as well as a quality one. Retrieved documents, web pages and tool output are data. They should be delimited, labelled as data, and never concatenated into the instruction section. The model reads one token stream; the structure you impose is the only separation it has.
Quick facts
- Scope
- The whole input, not the wording
- Core moves
- Select, compress, order, isolate
- Bound
- Token budget
- Evidence
- Ablation on an eval set
- Common failure
- Unowned growth
- Security boundary
- Trust labels and tool scopes
Composition as a function
A context is a function of the request and the available sources:
where is the candidate set, the query, and the token budget. Two systems with the same model and prompt can differ entirely in and in the selection policy. That is why copying a prompt rarely reproduces a product's quality.
The measurement that matters is marginal: for each section, what happens to task success when it is removed? Sections that never change the outcome are cost. Sections that change it only when combined with another are the ones a naive ablation will miss, so test combinations for the critical path.
Assembling a context
from dataclasses import dataclass
@dataclass
class Part:
name: str
text: str
priority: int # lower is more important
trust: str # "trusted" | "untrusted"
digest: str # content hash for dedupe
def assemble(parts, budget, count, clip):
seen, kept, used = set(), [], 0
for part in sorted(parts, key=lambda p: p.priority):
if part.digest in seen:
continue
seen.add(part.digest)
room = budget - used
if room <= 0:
break
tokens = count(part.text)
text = part.text if tokens <= room else clip(part.text, room)
kept.append({"name": part.name, "text": text, "trust": part.trust})
used += count(text)
return kept, {"used": used, "remaining": budget - used}Log the returned metadata with the request. When an answer is wrong, the first artefact to inspect is the assembled context: which parts were included, at what size, in what order, and what was dropped.
Trade-offs
| Strategy | Good for | Cost | Fails when |
|---|---|---|---|
| Full history in context | Short sessions, continuity | Linear token growth, distraction | Conversations run long or span topics |
| Summarised history | Long chat, bounded cost | Lossy, extra model call | Exact commitments or identifiers matter |
| Retrieval-selected context | Large corpora, changing knowledge | Retrieval latency and tuning | Recall is poor or permissions are complex |
| Static curated context | Stable policy and format | Manual maintenance | The domain changes faster than the curation |
| Long-context model, everything in | One-off analysis of a large artefact | Cost, latency, middle neglect | Requests repeat at volume |
Good for
Short sessions, continuity
Cost
Linear token growth, distraction
Fails when
Conversations run long or span topics
- Conversations run long or span topics
- Exact commitments or identifiers matter
- Recall is poor or permissions are complex
- The domain changes faster than the curation
- Requests repeat at volume
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full history in context is good for Short sessions, continuity, costs Linear token growth, distraction, and fails when Conversations run long or span topics. Summarised history is good for Long chat, bounded cost, costs Lossy, extra model call, and fails when Exact commitments or identifiers matter. Retrieval-selected context is good for Large corpora, changing knowledge, costs Retrieval latency and tuning, and fails when Recall is poor or permissions are complex. Static curated context is good for Stable policy and format, costs Manual maintenance, and fails when The domain changes faster than the curation. Long-context model, everything in is good for One-off analysis of a large artefact, costs Cost, latency, middle neglect, and fails when Requests repeat at volume
Failure modes
Context poisoning. A wrong fact enters memory or a summary and is re-injected on every turn. Debug by tracing the provenance of each context part and testing whether removing the suspect part changes the answer.
Distraction. Irrelevant but topical passages pull the answer off target. Debug by comparing accuracy at different retrieval depths; if accuracy falls as depth rises, the selector is admitting noise.
Context clash. Two sources disagree and the model averages them. Debug by looking for near-duplicate documents with different dates or versions, and resolve at selection time rather than asking the model to adjudicate.
Stale memory. A stored preference or summary outlives its truth. Debug by timestamping memory entries and testing expiry; memory without a freshness policy becomes a liability.
Symptom
A wrong fact enters memory or a summary and is re-injected on every turn.
Debug step
Tracing the provenance of each context part and testing whether removing the suspect part changes the answer.
Figure. Each failure with its symptom and the check that would confirm it.
What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.
Text version
Failure modes: Context poisoning: A wrong fact enters memory or a summary and is re-injected on every turn. Debug by tracing the provenance of each context part and testing whether removing the suspect part changes the answer. | Distraction: Irrelevant but topical passages pull the answer off target. Debug by comparing accuracy at different retrieval depths; if accuracy falls as depth rises, the selector is admitting noise. | Context clash: Two sources disagree and the model averages them. Debug by looking for near-duplicate documents with different dates or versions, and resolve at selection time rather than asking the model to adjudicate. | Stale memory: A stored preference or summary outlives its truth. Debug by timestamping memory entries and testing expiry; memory without a freshness policy becomes a liability.
In production
Track tokens per section, not just per request, and alert when a section's share grows without a corresponding quality gain. Cache the stable prefix. Hash the assembled context and store it with the trace so a bad answer is reproducible. Run the eval set against context variants — with and without history, retrieval depth 5 versus 15 — because context changes are behaviour changes. Keep an owner for the composition layer; the failure mode is nobody's job.
Common misconceptions
People think: Context engineering is prompt engineering with a new name.
Actually: Prompt engineering shapes instructions. Context engineering decides what evidence, tools, memory and history are present at all, which is often the larger effect.
People think: More context is safer because the model can ignore what it does not need.
Actually: Models do not reliably ignore irrelevant content. Long inputs cost more, run slower, and can reduce accuracy when the relevant passage is buried.
People think: Compaction is a lossless way to keep a conversation going.
Actually: Summaries drop detail. Commitments, identifiers and decisions must be preserved explicitly, or the compacted context will contradict the original.
Interview angle
A common prompt is "quality dropped after we added memory". The strong answer inspects the assembled context, ablates sections, checks ordering and trust labels, and proposes a budget with an owner. The weak answer rewrites the prompt. See the prompting bank and the system design bank.
Sources
- Anthropic, Effective context engineering for AI agents — selection, compression and context management for agents.
- Anthropic, Building effective agents — why context and workflow shape matter more than framework choice.
- Liu et al., Lost in the Middle — position effects in long contexts.
- Levy et al., Same Task, More Tokens — irrelevant input length degrades reasoning.
Related
- Context budget — the allocation arithmetic.
- Context selection — choosing what enters.
- Context compression — shrinking without losing the answer.
- Context ordering — position as a design variable.
- Memory versus context — what persists across turns.
- Untrusted input — the security side of composition.