Why it matters
A context window is the only working memory the model has for a request. Everything the application wants the model to consider must fit inside it, alongside everything the model needs in order to respond. In practice, that produces a competition: retrieved documents, conversation history, tool definitions and instructions all consume the same finite resource, and the output needs space too.
Most context bugs are allocation bugs. A prompt grows by one more retrieved chunk per release until the JSON answer is cut off. A chat product keeps the full transcript until one long support thread overflows. A tool-using agent adds schemas for every tool it might call and leaves too little room for the observation it just received. None of these are model failures; they are budgets without an owner.
Mental model
Packing a single desk for a day of work. You have a fixed surface, some items are mandatory (the brief, the tools), some are reference material, and you need empty space to actually write. Adding more paper does not help once the surface is full, and a stack of tangentially relevant reports can bury the one page you need.
The analogy breaks because attention is not uniform access. A document at the edge of the context is not simply "on the desk"; models tend to use the beginning and end of long inputs more reliably than the middle, and irrelevant material can actively distract. Position and salience matter as much as size.
How it works
- 1Reserve output tokens for the longest expected response, plus a safety margin.
- 2Charge fixed costs: system instructions, role framing, tool schemas and output-format rules.
- 3Rank candidate content — retrieved passages, memory, history — by expected value for this request.
- 4Select until the remaining budget is spent, with per-source caps so one source cannot dominate.
- 5Compress or drop low-value content, preferring extraction over lossy summarisation.
- 6Order what survives: stable instructions at the start, key evidence near the end, and no untrusted text in instruction position.
- 7Count the final rendered prompt, verify it fits, and log the utilisation.
Context window · illustrative tokens
1,800 input+ 1,000 reserve= 2,800 of 8,000
5,200 free
Step 1Reserve the answer
The 1,000-token output reserve is taken before any variable content. Only the fixed costs are charged so far, so 5,200 tokens remain for evidence and history.
Output reserve
1,000Pick a section to read its token count, its role and its state. Drag the window control to see what the budget does to the composition.
Figure. The output reserve comes off the top; instructions, tools, history and retrieval are then charged in order. When one more block consumes the reserve, the answer is cut off — the failure is arithmetic, not model quality.
What to notice. 8,000 tokens is smaller than current flagship windows, chosen so the allocation is visible. The counts are illustrative; the arithmetic inside the figure is exact for the numbers shown. The slider changes the window for the current step only.
Text version
An 8,000-token window with a 1,000-token output reserve. The default first step charges only the fixed costs — instructions 700, tool schemas 900 and the user request 200 — for 1,800 input tokens and 5,200 free. The second step adds 2,200 tokens of history. The third adds 2,800 tokens of retrieved passages, leaving 200 free. The fourth replaces history with a 3,400-token long thread: input reaches 8,000, the reserve is consumed, and the window is 1,000 tokens over. The fifth step compacts history to 900 tokens and restores 1,500 tokens of headroom. A window slider runs from 5,000 to 9,000 tokens, and selecting any block shows its count and role.
The first two steps are non-negotiable. Output reservation is the one budget line that teams routinely forget, and it produces the most confusing symptom: an answer that starts correctly and ends in the middle of a JSON object. Tool schemas are the second hidden cost; an agent with twenty tools can spend a large fraction of its window describing them before it has read anything.
Selection is where quality is won. A retrieval system that returns ten chunks regardless of relevance is not a budget strategy. Rank candidates, apply per-source caps, and require a minimum score so that weak evidence is dropped rather than padded in. When a section must shrink, prefer extracting the sentences that answer the question over summarising the whole document: summaries introduce another model's errors and lose exact identifiers.
Ordering is the cheapest improvement available. Put stable instructions at the start so they can be cached, and put the decisive evidence close to the generation point rather than in the middle of a long block. If the prompt contains untrusted content, keep it in a clearly delimited data section and never adjacent to instruction text without a boundary.
The budget equation
For a single request:
where is the window, is the reserved output, and the rest are the sections you assembled. Utilisation is the sum of the input terms divided by . High utilisation is not a goal; it is a warning that the next longer input will fail.
Marginal value is the concept that keeps the budget honest. The hundredth retrieved chunk is worth less than the fifth, and the last thousand tokens of chat history are usually worth less than the current question's evidence. Spend where the expected improvement per token is highest, and measure the effect rather than assuming it.
Planning the budget in code
from dataclasses import dataclass
@dataclass
class Section:
name: str
text: str
priority: int # lower wins when the budget is tight
cap: int | None = None # per-source ceiling in tokens
def plan_context(sections, window, reserve, count):
budget = window - reserve
plan, spent = [], 0
for section in sorted(sections, key=lambda s: s.priority):
tokens = count(section.text)
ceiling = section.cap or tokens
take = min(tokens, ceiling, budget - spent)
if take <= 0:
continue
plan.append({
"name": section.name,
"tokens": take,
"truncated": take < tokens,
})
spent += take
return {"plan": plan, "used": spent, "remaining": budget - spent}The returned plan is the artefact to log. When an answer is wrong, the first question is which sections were included, at what size, and what was dropped. A plan makes that answerable without replaying the request.
Text version
Starting from the model window, the fixed system prompt and capped history are allocated, retrieval fills the remainder, and the output reserve stays untouched.
Trade-offs
| Strategy | Good for | Cost | Fails when |
|---|---|---|---|
| Stuff the full window | Short, high-recall tasks | Highest token cost and latency | Evidence is noisy or the task needs precision |
| Fixed top-k retrieval | Simple, predictable pipelines | Wastes budget on weak hits | Relevance varies widely per query |
| Adaptive selection with caps | Mixed corpora, production chat | Needs scoring and tuning | Scores are uncalibrated across sources |
| Summarise history to fit | Long conversations | Lossy, adds a model call | Exact numbers, IDs or commitments matter |
| Compact and restart | Agent loops with long traces | Loses detail, changes behaviour | The task depends on earlier observations |
Good for
Short, high-recall tasks
Cost
Highest token cost and latency
Fails when
Evidence is noisy or the task needs precision
- Evidence is noisy or the task needs precision
- Relevance varies widely per query
- Scores are uncalibrated across sources
- Exact numbers, IDs or commitments matter
- The task depends on earlier observations
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Stuff the full window is good for Short, high-recall tasks, costs Highest token cost and latency, and fails when Evidence is noisy or the task needs precision. Fixed top-k retrieval is good for Simple, predictable pipelines, costs Wastes budget on weak hits, and fails when Relevance varies widely per query. Adaptive selection with caps is good for Mixed corpora, production chat, costs Needs scoring and tuning, and fails when Scores are uncalibrated across sources. Summarise history to fit is good for Long conversations, costs Lossy, adds a model call, and fails when Exact numbers, IDs or commitments matter. Compact and restart is good for Agent loops with long traces, costs Loses detail, changes behaviour, and fails when The task depends on earlier observations
Failure modes
Output truncated at the cap. The prompt consumed the space reserved for the answer. Debug by logging prompt tokens, reserved output tokens and the finish reason together; if the finish reason is a length stop, the reserve was too small.
Silent provider-side truncation. Some paths drop the oldest content rather than erroring. Debug by comparing the tokens you sent with the provider's reported input count, and by asserting that a unique marker at the start and end of the prompt survives into the rendered context.
History crowds out evidence. A long conversation keeps its full transcript and retrieval gets whatever remains. Debug by plotting tokens per section over turns; if history grows monotonically, introduce a compaction policy with an explicit cap.
Middle-of-context neglect. The decisive passage is present but the answer ignores it. Debug by moving the passage to the end of the evidence block and re-running; if the answer changes, the problem is ordering, not retrieval.
In production
Keep the stable prefix cacheable and the variable content after it. Alert on utilisation above a threshold and on the share of requests that hit the output cap. Track dropped sections per request, because a sudden rise is an early signal that the corpus or the conversation shape changed. Re-run the eval set whenever the allocation policy changes: context changes are model-visible behaviour changes, even when the prompt wording is untouched.
Common misconceptions
People think: A large context window removes the need for selection.
Actually: A larger window raises the ceiling, not the quality per token. Long inputs cost more, run slower, and can reduce accuracy when relevant content is buried.
People think: If the prompt fits, the request is safe.
Actually: The output needs room too. A full prompt plus a long answer is the standard recipe for truncation.
Interview angle
Context budget questions test whether you can reason about a finite resource with competing consumers. Strong answers reserve output, name the fixed costs, describe a ranking policy, and give a metric that detects drift. Weak answers quote the window size as if it were available capacity. See the prompting questions and the token budget drill.
Sources
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts — position effects in long inputs.
- Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models? — effective context versus advertised context.
- Levy et al., Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of LLMs — irrelevant length degrades reasoning.
- Anthropic, Context windows — how the window and prompt structure are documented for a hosted model.
Related
- Tokens — the unit the budget is denominated in.
- Context engineering — the broader discipline.
- Long-context trade-offs — when a bigger window helps.
- Context compression — shrinking without losing the answer.
- Prompt caching — making the stable prefix cheap.