Why it matters
Context limits are enforced by truncation somewhere in the stack, often at the last moment and in arrival order. That means a long retrieved document can push out the system instruction or the most recent user turn. Prioritisation makes the ordering explicit before the request is built, so the material that matters survives, and the material that is dropped is recorded and can be reviewed.
Mental model
Token prioritisation is triage at a hospital: the most urgent cases are treated first, not the ones that arrived first. The comparison breaks because urgency here is estimated by a scorer, and a mis-scored case can wait unnoticed. There is no clinician walking the queue.
How it works
Fill order · highest priority first · illustrative tokens
4,150 input+ 800 reserve= 4,950 of 5,000
50 free
Step 1Fixed costs first
Instructions, tool schemas, the current turn and the 800-token reserve are charged first; what remains is spent on the variable blocks in priority order, and the tail falls out when it runs out.
Output reserve
800Pick a section to read its token count, its role and its state. Drag the window control to see what the budget does to the composition.
Not in the prompt · 2
Select an entry to read why it was left out.
Figure. The window is filled in priority order; when it shrinks, the lowest-value unpinned blocks fall out first while instructions, tools and the current turn stay. The squeeze control changes the window for the current step.
What to notice. The ranking is a policy chosen for this example, not a universal ordering. In a real system value per token is estimated by a scorer; here the counts and the ranking are authored, and no model was measured.
Text version
A 5,000-token window with an 800-token output reserve. Nine blocks sit in priority order: system instructions 600, tool schemas 500 and current user turn 150 are pinned; key evidence 900, recent history 800, an extra retrieved document 700, few-shot examples 500, older history 600 and a background note 400 are not. The default first step has 4,150 input tokens plus the reserve, 50 free, with background and older history outside the window. The second step at a 4,000-token window also drops the extra document and the examples. The third step at 2,800 leaves only the pinned core, because the key evidence no longer fits. A squeeze slider runs from 2,400 to 6,000 tokens. Selecting any block shows its token count, kind and why it ranks where it does.
- Compute fixed costs: system instructions, tool schemas, template overhead and the output reserve.
- Assign each remaining item a category with a priority: current user turn, retrieved evidence, recent history, examples, background.
- Estimate tokens per item and value per item, then sort within category by value per token.
- Fill the remaining budget in priority order and stop at the boundary rather than truncating mid-item.
- Record which items were dropped and why, and expose the count in logs.
- Validate the policy by removing the lowest-priority category and measuring the quality change.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Strict priority order | Protects instructions and evidence | May drop useful background | The task needs broad context |
| Equal share per category | Predictable | Wastes budget on empty categories | One source dominates the request |
| Value-per-token ranking | Efficient use of budget | Scoring errors drop important items | Value is hard to estimate |
| Truncate at the end | Simple | Cuts the newest or most relevant material | The prompt is assembled in variable order |
Good for
Protects instructions and evidence
Cost
May drop useful background
Fails when
The task needs broad context
- The task needs broad context
- One source dominates the request
- Value is hard to estimate
- The prompt is assembled in variable order
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Strict priority order is good for Protects instructions and evidence, costs May drop useful background, and fails when The task needs broad context. Equal share per category is good for Predictable, costs Wastes budget on empty categories, and fails when One source dominates the request. Value-per-token ranking is good for Efficient use of budget, costs Scoring errors drop important items, and fails when Value is hard to estimate. Truncate at the end is good for Simple, costs Cuts the newest or most relevant material, and fails when The prompt is assembled in variable order
Failure modes
- Instruction truncation. A long document pushes the system prompt out of the window. Debug: assert that fixed costs are subtracted before any variable content is added.
- Mid-item cut. A passage is truncated mid-sentence and the model completes the thought incorrectly. Debug: check that truncation happens at item boundaries.
- Silent drop. The evidence was excluded and no one knows why the answer is vague. Debug: log excluded item ids with the request id and review them for failures.
In production
Monitor the dropped-item rate and the token share of each category. Alert when the output reserve is consumed by input growth, since that is where truncation starts.
Common misconceptions
People think: Truncation is a rare edge case.
Actually: Long conversations and large documents hit the limit routinely, often on the requests that matter most.
People think: Dropping the oldest content is always safe.
Actually: Early turns often contain constraints and identifiers that later turns refer to.
Interview angle
Show the budget arithmetic, then defend the priority order and the item-boundary rule. Strong answers log what was dropped and evaluate the policy. Weak answers truncate by character count at the end and have never checked which instruction was lost.
Sources
- Liu et al., Lost in the Middle — accuracy depends on where material sits, not only on whether it fits.
- Jiang et al., LLMLingua — token-level compression when the budget cannot hold everything.
- Anthropic, Effective context engineering for AI agents — budgeting context as a finite resource.