Why it matters
Every request is a budget negotiation. The system prompt, conversation history, retrieved documents, tool definitions and the model's own output all draw from one limit. When the budget is exceeded, something gets cut — and whatever gets cut is usually decided by code written months earlier, not by the person who understands the query.
The window also drives cost and latency in ways that are easy to miss. A long prompt is processed before the first token appears, so time-to-first-token tracks prompt length. The same long prompt is re-read on every decode step through the cache, so throughput tracks both lengths. Teams that treat the window as "free space" discover this on the first invoice.
Mental model: the size of the desk
Think of the window as the surface area of a desk. Everything the model can look at during one task must be on the desk at once: the instructions, the documents, the conversation so far, and the notepad where it writes the answer. If the desk is full, something has to come off before work continues.
The analogy is good for budgeting: you reserve space for the answer before you pile on documents.
It breaks because a desk does not care where things are placed, and a transformer does. Evidence near the start and end of a long prompt is used more reliably than evidence buried in the middle. The desk is also not uniformly readable: attention is spread unevenly, so "it fits" and "it is used" are different claims.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- window
- 8192 tokens
- request A headroom
- 8192 − 8192 = 0
- request B headroom
- 8192 − 7400 = 792
- request C headroom
- 8192 − 4300 = 3892
Step 1 of 4
Three requests, one window
Each row is a full request: the prompt parts plus the output the caller reserved. The row total is the number that has to stay at or below 8,192, not the prompt alone.
Next: Request A is full
Figure. Three requests against one 8,192-token window, illustrative counts. Rows are requests, columns the parts that draw from the budget — a fixed prefix, documents, history and reserved output — and a row total is prompt plus output. Request A fills the window exactly at 8,192; B leaves 792 tokens unused; C leaves 3,892. The output reserve is a debit even though those tokens are generated later.
What to notice. Read the columns as well as the rows: the fixed prefix costs 1,400 tokens in requests A and B and 1,100 in C, where fewer tools are attached, while documents range from 1,600 to 3,792. A row total equal to the window means no headroom, so one more retrieved document has to displace something. The column totals add three independent requests, so they are a workload view and not one budget; no single call can use 8,492 document tokens.
Text version
A three by four table. Rows are request A, request B and request C; columns are the fixed prefix, docs, history and output, counted in tokens. Request A prints 1,400, 3,792, 1,400 and 1,600 and totals 8,192. Request B prints 1,400, 3,100, 1,300 and 1,600 and totals 7,400. Request C prints 1,100, 1,600, 800 and 800 and totals 4,300. Column totals read 3,900, 8,492, 3,500 and 4,000. Steps highlight request A, the output column and request C; the readout reports the request, the budget part and the token count with the row total.
- Count in tokens. Convert the prompt with the model's own tokenizer. Words, characters and bytes are all misleading units here.
- Reserve output space. Set
max_output_tokensexplicitly and subtract it before assembling context. A request that fills the window leaves no room to answer. - Budget the fixed parts. System prompt, tool schemas, chat template and any always-on instructions are spent before retrieval. Measure them once and treat changes as budget changes.
- Fill with variable context. Allocate the remainder to retrieved documents, history or attachments, ranked by expected value rather than by recency alone.
- Apply a policy when over budget. Drop lowest-value items, summarise history, or fail the request. Silent truncation at a raw token index is the worst option because it removes structure as well as content.
- Send and record. Log prompt tokens, output tokens, whether truncation fired and which policy ran. These are the metrics that explain later incidents.
- Extend with care. Position interpolation, rotary scaling and attention variants can stretch a trained window, but extension changes behaviour and needs evaluation on your task, not just on a length benchmark.
- Unit
- tokens, not characters or words
- Includes
- prompt + generated output
- Cost shape
- prefill linear in prompt, attention quadratic
- Effective length
- often below the advertised limit
The math
The hard constraint for a single request:
Prefill compute scales approximately with parameters times prompt tokens, and the attention score matrix scales with the square of sequence length:
The first term explains why long prompts cost more; the second explains why very long prompts cost disproportionately more and why memory becomes the binding constraint before compute does.
A minimal implementation
def plan_context(
tokenizer,
system: str,
documents: list[str],
question: str,
window: int,
max_output: int = 1024,
safety: int = 64,
) -> tuple[list[str], dict]:
"""Keep highest-priority documents that fit; report what was dropped."""
fixed = len(tokenizer.encode(system)) + len(tokenizer.encode(question))
budget = window - max_output - safety - fixed
if budget <= 0:
raise ValueError("system prompt and question exceed the window")
kept, used = [], 0
for doc in documents: # already ranked by relevance
n = len(tokenizer.encode(doc))
if used + n > budget:
continue # skip, do not truncate mid-document
kept.append(doc)
used += n
return kept, {"used": used, "budget": budget, "dropped": len(documents) - len(kept)}Skipping an oversized document rather than cutting it is deliberate: half a policy page is worse evidence than the next-best complete one.
Trade-offs
| Strategy | Good for | Cost | Fails when |
|---|---|---|---|
| Small window + retrieval | Large corpora, fresh facts, attributable answers | Retrieval pipeline and its failures | The task needs the whole document to reason |
| Stuff the long window | Whole-document analysis, small corpora | Prefill latency and per-call price | Most of the document is irrelevant or evidence is mid-context |
| Sliding window over chunks | Streaming, very long inputs | Loses cross-chunk context | The answer depends on distant sections |
| Hierarchical summarisation | Long histories, reports | Summaries lose detail recursively | Exact quotes or numbers are required |
| History compaction | Chat products with long sessions | Extra model calls, drift | The user refers back to an early detail |
Good for
Large corpora, fresh facts, attributable answers
Cost
Retrieval pipeline and its failures
Fails when
The task needs the whole document to reason
- The task needs the whole document to reason
- Most of the document is irrelevant or evidence is mid-context
- The answer depends on distant sections
- Exact quotes or numbers are required
- The user refers back to an early detail
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Small window + retrieval is good for Large corpora, fresh facts, attributable answers, costs Retrieval pipeline and its failures, and fails when The task needs the whole document to reason. Stuff the long window is good for Whole-document analysis, small corpora, costs Prefill latency and per-call price, and fails when Most of the document is irrelevant or evidence is mid-context. Sliding window over chunks is good for Streaming, very long inputs, costs Loses cross-chunk context, and fails when The answer depends on distant sections. Hierarchical summarisation is good for Long histories, reports, costs Summaries lose detail recursively, and fails when Exact quotes or numbers are required. History compaction is good for Chat products with long sessions, costs Extra model calls, drift, and fails when The user refers back to an early detail
Failure modes
The newest instruction is truncated. A naive tokens[:limit] keeps the oldest content and drops the question. Debugging step: log the last 200 characters of every assembled prompt and alert when the user's message is not present.
Counting with the wrong tokenizer. The budget is computed with a local estimate, but the provider tokenizes differently. Debugging step: compare your count against the provider's reported prompt tokens on the first N requests; a constant offset means a template or tokenizer mismatch.
Advertised length treated as effective length. A 200k window is used to justify dropping retrieval, and answer quality falls. Debugging step: evaluate accuracy by evidence position — start, middle, end — at several context lengths, on your own task.
Cost per request creeps with history. Conversations grow until every turn re-sends a large transcript. Debugging step: chart prompt tokens per request over time; a monotonic rise with session length is the signal to compact or summarise.
In production
Monitor prompt tokens at p50 and p95, truncation rate, and the fraction of the window used. These three metrics explain most latency and cost surprises better than aggregate token spend.
Prefer retrieval over stuffing when the corpus is large and changing. A bigger window does not remove the need to select: it changes the failure from "not retrieved" to "retrieved but ignored", which is harder to debug.
Use prompt caching for the stable prefix — system prompt and tool schemas — and keep those sections byte-identical. Cache hit rate is a context-design metric: it tells you how much of your prompt is genuinely reusable.
Common misconceptions
People think: A 200k-token window means the model can use 200k tokens of evidence.
Actually: It can accept them. Utilisation is uneven, and accuracy on evidence in the middle of a long context is typically lower than at the edges.
People think: The context window is a memory limit for the model.
Actually: It is a limit on positions the model can condition on in one pass. Memory during serving is a separate, cache-driven constraint.
A third: that a larger window removes the need for chunking and retrieval. It changes the engineering problem rather than deleting it.
Interview angle
Context questions are budget questions with a quality twist. Be ready to compute a budget from a window, reserve output tokens, and explain what changes in cost when the prompt doubles. Then answer the harder part: why a larger window does not guarantee better answers, and what you would measure to find out. See context budget and long-context attention, plus the RAG questions at /interview/rag.
Sources
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts — position-dependent degradation in long prompts: https://arxiv.org/abs/2307.03172
- Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding — the positional scheme most long-context work builds on: https://arxiv.org/abs/2104.09864
- Chen et al., Extending Context Window of Large Language Models via Positional Interpolation — one method for stretching a trained window: https://arxiv.org/abs/2306.15595
- Pope et al., Efficiently Scaling Transformer Inference — how prefill and decode cost scale with length: https://arxiv.org/abs/2211.05102