Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 26 of 33

Context window

0 of 33 complete

Why it matters

Every request is a budget negotiation. The system prompt, conversation history, retrieved documents, tool definitions and the model's own output all draw from one limit. When the budget is exceeded, something gets cut — and whatever gets cut is usually decided by code written months earlier, not by the person who understands the query.

The window also drives cost and latency in ways that are easy to miss. A long prompt is processed before the first token appears, so time-to-first-token tracks prompt length. The same long prompt is re-read on every decode step through the cache, so throughput tracks both lengths. Teams that treat the window as "free space" discover this on the first invoice.

Mental model: the size of the desk

Think of the window as the surface area of a desk. Everything the model can look at during one task must be on the desk at once: the instructions, the documents, the conversation so far, and the notepad where it writes the answer. If the desk is full, something has to come off before work continues.

The analogy is good for budgeting: you reserve space for the answer before you pile on documents.

It breaks because a desk does not care where things are placed, and a transformer does. Evidence near the start and end of a long prompt is used more reliably than evidence buried in the middle. The desk is also not uniformly readable: attention is spread unevenly, so "it fits" and "it is used" are different claims.

How it works

One window · where three requests spend itInteractive

hover or focus a cell

03792
request ↓budget part →
fixeddocshistoryoutputtokens
request A8192
request B7400
request C4300
total3900849235004000

Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.

window
8192 tokens
request A headroom
8192 − 8192 = 0
request B headroom
8192 − 7400 = 792
request C headroom
8192 − 4300 = 3892

Step 1 of 4

Three requests, one window

Each row is a full request: the prompt parts plus the output the caller reserved. The row total is the number that has to stay at or below 8,192, not the prompt alone.

Next: Request A is full

Step 1/4 · Three requests, one window

Figure. Three requests against one 8,192-token window, illustrative counts. Rows are requests, columns the parts that draw from the budget — a fixed prefix, documents, history and reserved output — and a row total is prompt plus output. Request A fills the window exactly at 8,192; B leaves 792 tokens unused; C leaves 3,892. The output reserve is a debit even though those tokens are generated later.

What to notice. Read the columns as well as the rows: the fixed prefix costs 1,400 tokens in requests A and B and 1,100 in C, where fewer tools are attached, while documents range from 1,600 to 3,792. A row total equal to the window means no headroom, so one more retrieved document has to displace something. The column totals add three independent requests, so they are a workload view and not one budget; no single call can use 8,492 document tokens.

Text version

A three by four table. Rows are request A, request B and request C; columns are the fixed prefix, docs, history and output, counted in tokens. Request A prints 1,400, 3,792, 1,400 and 1,600 and totals 8,192. Request B prints 1,400, 3,100, 1,300 and 1,600 and totals 7,400. Request C prints 1,100, 1,600, 800 and 800 and totals 4,300. Column totals read 3,900, 8,492, 3,500 and 4,000. Steps highlight request A, the output column and request C; the readout reports the request, the budget part and the token count with the row total.

  1. Count in tokens. Convert the prompt with the model's own tokenizer. Words, characters and bytes are all misleading units here.
  2. Reserve output space. Set max_output_tokens explicitly and subtract it before assembling context. A request that fills the window leaves no room to answer.
  3. Budget the fixed parts. System prompt, tool schemas, chat template and any always-on instructions are spent before retrieval. Measure them once and treat changes as budget changes.
  4. Fill with variable context. Allocate the remainder to retrieved documents, history or attachments, ranked by expected value rather than by recency alone.
  5. Apply a policy when over budget. Drop lowest-value items, summarise history, or fail the request. Silent truncation at a raw token index is the worst option because it removes structure as well as content.
  6. Send and record. Log prompt tokens, output tokens, whether truncation fired and which policy ran. These are the metrics that explain later incidents.
  7. Extend with care. Position interpolation, rotary scaling and attention variants can stretch a trained window, but extension changes behaviour and needs evaluation on your task, not just on a length benchmark.
Unit
tokens, not characters or words
Includes
prompt + generated output
Cost shape
prefill linear in prompt, attention quadratic
Effective length
often below the advertised limit

The math

The hard constraint for a single request:

nsystem+ncontext+noutput≤nwindow.n_{\text{system}} + n_{\text{context}} + n_{\text{output}} \le n_{\text{window}}.

Prefill compute scales approximately with parameters times prompt tokens, and the attention score matrix scales with the square of sequence length:

prefill FLOPs≈2 P nprompt,attention scores=O ⁣(nprompt2).\text{prefill FLOPs} \approx 2\,P\,n_{\text{prompt}}, \qquad \text{attention scores} = O\!\left(n_{\text{prompt}}^2\right).

The first term explains why long prompts cost more; the second explains why very long prompts cost disproportionately more and why memory becomes the binding constraint before compute does.

A minimal implementation

def plan_context(
    tokenizer,
    system: str,
    documents: list[str],
    question: str,
    window: int,
    max_output: int = 1024,
    safety: int = 64,
) -> tuple[list[str], dict]:
    """Keep highest-priority documents that fit; report what was dropped."""
    fixed = len(tokenizer.encode(system)) + len(tokenizer.encode(question))
    budget = window - max_output - safety - fixed
    if budget <= 0:
        raise ValueError("system prompt and question exceed the window")
 
    kept, used = [], 0
    for doc in documents:                       # already ranked by relevance
        n = len(tokenizer.encode(doc))
        if used + n > budget:
            continue                            # skip, do not truncate mid-document
        kept.append(doc)
        used += n
 
    return kept, {"used": used, "budget": budget, "dropped": len(documents) - len(kept)}

Skipping an oversized document rather than cutting it is deliberate: half a policy page is worse evidence than the next-best complete one.

Trade-offs

StrategyGood forCostFails when
Small window + retrievalLarge corpora, fresh facts, attributable answersRetrieval pipeline and its failuresThe task needs the whole document to reason
Stuff the long windowWhole-document analysis, small corporaPrefill latency and per-call priceMost of the document is irrelevant or evidence is mid-context
Sliding window over chunksStreaming, very long inputsLoses cross-chunk contextThe answer depends on distant sections
Hierarchical summarisationLong histories, reportsSummaries lose detail recursivelyExact quotes or numbers are required
History compactionChat products with long sessionsExtra model calls, driftThe user refers back to an early detail
Trade-off mapInteractive

Good for

Large corpora, fresh facts, attributable answers

Cost

Retrieval pipeline and its failures

Fails when

The task needs the whole document to reason

  1. The task needs the whole document to reason
  2. Most of the document is irrelevant or evidence is mid-context
  3. The answer depends on distant sections
  4. Exact quotes or numbers are required
  5. The user refers back to an early detail

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Small window + retrieval is good for Large corpora, fresh facts, attributable answers, costs Retrieval pipeline and its failures, and fails when The task needs the whole document to reason. Stuff the long window is good for Whole-document analysis, small corpora, costs Prefill latency and per-call price, and fails when Most of the document is irrelevant or evidence is mid-context. Sliding window over chunks is good for Streaming, very long inputs, costs Loses cross-chunk context, and fails when The answer depends on distant sections. Hierarchical summarisation is good for Long histories, reports, costs Summaries lose detail recursively, and fails when Exact quotes or numbers are required. History compaction is good for Chat products with long sessions, costs Extra model calls, drift, and fails when The user refers back to an early detail

Failure modes

The newest instruction is truncated. A naive tokens[:limit] keeps the oldest content and drops the question. Debugging step: log the last 200 characters of every assembled prompt and alert when the user's message is not present.

Counting with the wrong tokenizer. The budget is computed with a local estimate, but the provider tokenizes differently. Debugging step: compare your count against the provider's reported prompt tokens on the first N requests; a constant offset means a template or tokenizer mismatch.

Advertised length treated as effective length. A 200k window is used to justify dropping retrieval, and answer quality falls. Debugging step: evaluate accuracy by evidence position — start, middle, end — at several context lengths, on your own task.

Cost per request creeps with history. Conversations grow until every turn re-sends a large transcript. Debugging step: chart prompt tokens per request over time; a monotonic rise with session length is the signal to compact or summarise.

In production

Monitor prompt tokens at p50 and p95, truncation rate, and the fraction of the window used. These three metrics explain most latency and cost surprises better than aggregate token spend.

Prefer retrieval over stuffing when the corpus is large and changing. A bigger window does not remove the need to select: it changes the failure from "not retrieved" to "retrieved but ignored", which is harder to debug.

Use prompt caching for the stable prefix — system prompt and tool schemas — and keep those sections byte-identical. Cache hit rate is a context-design metric: it tells you how much of your prompt is genuinely reusable.

Common misconceptions

People think: A 200k-token window means the model can use 200k tokens of evidence.

Actually: It can accept them. Utilisation is uneven, and accuracy on evidence in the middle of a long context is typically lower than at the edges.

People think: The context window is a memory limit for the model.

Actually: It is a limit on positions the model can condition on in one pass. Memory during serving is a separate, cache-driven constraint.

A third: that a larger window removes the need for chunking and retrieval. It changes the engineering problem rather than deleting it.

Interview angle

Context questions are budget questions with a quality twist. Be ready to compute a budget from a window, reserve output tokens, and explain what changes in cost when the prompt doubles. Then answer the harder part: why a larger window does not guarantee better answers, and what you would measure to find out. See context budget and long-context attention, plus the RAG questions at /interview/rag.

Sources