Why it matters
Context windows are large enough to encourage waste and finite enough to fail at the worst moment. A request that fits in a demo can exceed the window with a long document or a long conversation, and the failure is usually a truncated answer or a dropped instruction. Designing a token budget means deciding the allocation before the request arrives and specifying which part is sacrificed when it does not fit. That decision is what the whiteboard question is testing.
Mental model
It is packing a suitcase with a hard weight limit: the heavy items are chosen first and something is left behind deliberately. The metaphor breaks because the limit is shared with the output, so packing too much input can prevent the answer from being written at all.
How it works
Context budget · illustrative tokens
800 input+ 1,000 reserve= 1,800 of 5,000
3,200 free
Step 1Reserve the answer
The 1,000-token output reserve comes off the top, then the fixed costs: system 300, tools 400 and question 100. That is 1,800 of 5,000, leaving 3,200 for evidence and history.
Output reserve
1,000Pick a section to read its token count, its role and its state.
Figure. The answer to the token-budget question is an allocation: reserve the output first, cap every input line item, and state what leaves when the window fills. Here two documents arrive at once and the lower-ranked one is cut, not squeezed.
What to notice. Counts are authored for a 5,000-token window with a 1,000-token output reserve; no model was called and no provider limit is claimed. The caps and the drop order are one defensible policy, not a universal one. Streaming, caching and per-provider pricing are omitted; the point is the allocation and the overflow path.
Text version
A 5,000-token window with a 1,000-token output reserve, so 4,000 tokens are available. The default first step charges only system and policy 300, tool definitions 400 and the current question 100, for 800 input tokens and 3,200 free. The second step adds worked examples 250, history 800 and retrieved evidence 1,500, for 3,350 tokens and 650 free. The third step adds a 900-token document and a second 600-token document at once: 4,850 input tokens, 850 over the available window, with the cells crossing the cap rule. The fourth step drops the examples first and then the lower-ranked second document, returning to a 4,000-token input that fills the window exactly and leaves the reserve intact. The fifth step names the drop order — examples, history, lower-ranked evidence — and says the instructions and output reserve are never in it. Selecting a line item shows its count, its cap and its role.
- Set the total window and reserve a fixed output allowance before allocating any input.
- Assign caps to each input line item: system instructions, few-shot examples, retrieved passages, conversation history and tool output.
- Rank retrieved passages and cut by rank when the evidence cap is exceeded, keeping citation ids for what remains.
- Summarise or window history when it passes its cap, and record what was dropped.
- Handle overflow with an explicit order and a user-visible signal rather than silent truncation.
- Measure the distribution of input tokens per request in production and set alerts near the cap.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Large evidence cap | More recall and grounding | Less room for history and output | Questions need conversational context |
| Large history cap | Better continuity | Stale turns crowd out evidence | The task is a fresh lookup each time |
| Fixed allocation | Predictable and simple | Wasteful for short requests | Traffic mixes short and long inputs |
| Dynamic allocation | Efficient use of the window | Harder to reason about and test | A bug in the allocator truncates output |
Good for
More recall and grounding
Cost
Less room for history and output
Fails when
Questions need conversational context
- Questions need conversational context
- The task is a fresh lookup each time
- Traffic mixes short and long inputs
- A bug in the allocator truncates output
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Large evidence cap is good for More recall and grounding, costs Less room for history and output, and fails when Questions need conversational context. Large history cap is good for Better continuity, costs Stale turns crowd out evidence, and fails when The task is a fresh lookup each time. Fixed allocation is good for Predictable and simple, costs Wasteful for short requests, and fails when Traffic mixes short and long inputs. Dynamic allocation is good for Efficient use of the window, costs Harder to reason about and test, and fails when A bug in the allocator truncates output
Failure modes
- Output truncation. Input fills the window and the answer stops mid-sentence. Debug: log input tokens plus max output against the model limit for failing requests.
- Dropped instruction. A late-arriving document pushes the system instruction out of a summarised history. Debug: check whether the instruction appears in the final assembled prompt for failing cases.
- Silent evidence cut. The passage that answers the question is dropped by rank and the model abstains. Debug: compare the retrieved ids with the ids present in the assembled prompt.
In production
Log token counts per line item and alert when a request uses more than a set share of the window. Keep the overflow path tested with a synthetic long input.
Common misconceptions
People think: A bigger context window removes the need for a budget.
Actually: It raises the ceiling and the cost; allocation and overflow behaviour still have to be designed.
People think: More retrieved context always improves answers.
Actually: Beyond the relevant evidence it adds cost and distracts, and it can push out instructions or output space.
Interview angle
Strong answers reserve output first, cap each line item and state the drop order and the signal. Weak answers say they would pass as much context as fits.
Sources
- Kwon et al., PagedAttention and vLLM — how context length drives memory and throughput.
- Vaswani et al., Attention Is All You Need — the attention cost that scales with sequence length.
- Anthropic, Prompt caching — how a stable prefix changes the cost of a budget.