Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 16 of 26

Context selection

0 of 26 complete

Why it matters

A context window is large but finite, and every token spent on low-value material is a token unavailable for evidence. Applications accumulate candidates from retrieval, conversation history and tool output, then include them in arrival order. Selection makes the choice explicit: mandatory content first, then scored candidates by value per token, with a reserve for the answer. Without it, the prompt grows until something important falls out.

Mental model

Context selection is packing a suitcase with a weight limit: take what the trip needs, not everything that might be useful. The comparison breaks because an extra item here can make the important ones harder to find, not just heavier.

How it works

Selection under a token ceilingInteractive

Selection budget · illustrative tokens

4,920 input+ 500 reserve= 5,420 of 4,000

over by 1,420

Step 1/3 · Everything offered

Step 1Everything offered

Arrival order is not a policy. The candidates total 4,920 tokens against 3,500 available, so something has to be left out.

reserve consumed4,920 tokens

Output reserve

500
output reservefixedBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. Selection is ranking under a hard ceiling: mandatory content is charged first, scored candidates fill the rest in value order, and the boundary stops at an item edge. What does not fit is recorded as excluded.

What to notice. Scores, token counts and the relevance floor are illustrative. The figure shows what fits and what is left out, not whether the ranking is correct for a real query.

Text version

A 4,000-token window with a 500-token output reserve. The default first step offers nine candidates in arrival order — instructions 500, history 900, three retrieved passages of 700, 500 and 900, a marketing FAQ 600, an order lookup 300, two worked examples 400 and the current question 120 — for 4,920 input tokens, 1,420 over the 3,500 available, shown as a hatched band over the reserve. The next step reorders them by value: mandatory instructions and question first, then the strongest passage, the second passage, the tool result and history, with the weakest candidates last. The third step keeps instructions, question, two passages, the tool result and 400 tokens of history; the 2023 policy, the marketing FAQ and the examples sit outside the window with reasons. Selecting any block shows its score, token count and role.

  1. Classify candidate material as mandatory, scored or disposable, and reserve output tokens first.
  2. Score evidence for relevance to the current request, and normalise by token cost.
  3. Fill the budget in priority order, stopping when the next candidate would displace something more valuable.
  4. Apply source-specific rules: keep recent turns, drop superseded documents, keep tool results only while relevant.
  5. Record what was included and excluded, with the reason, for each request.
  6. Evaluate the policy by dropping the lowest-ranked items and measuring whether quality moves.

Trade-offs

ChoiceGood forCostFails when
Relevance-ranked selectionHigh value per tokenRanking errors drop needed evidenceRelevance scores are unreliable for the domain
Recency-firstGood for conversationsLoses early commitmentsThe task depends on an old constraint
Include everything that fitsNo selection riskDilution and higher costThe window is large and the corpus noisy
Fixed allocation per sourcePredictable budgetsWastes space when a source is emptySource needs vary per request
Trade-off mapInteractive

Good for

High value per token

Cost

Ranking errors drop needed evidence

Fails when

Relevance scores are unreliable for the domain

  1. Relevance scores are unreliable for the domain
  2. The task depends on an old constraint
  3. The window is large and the corpus noisy
  4. Source needs vary per request

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Relevance-ranked selection is good for High value per token, costs Ranking errors drop needed evidence, and fails when Relevance scores are unreliable for the domain. Recency-first is good for Good for conversations, costs Loses early commitments, and fails when The task depends on an old constraint. Include everything that fits is good for No selection risk, costs Dilution and higher cost, and fails when The window is large and the corpus noisy. Fixed allocation per source is good for Predictable budgets, costs Wastes space when a source is empty, and fails when Source needs vary per request

Failure modes

  • Budget starvation. Mandatory instructions consume the window and evidence is truncated. Debug: log tokens per source category and check the reserve for output.
  • Ranking blind spot. The needed passage scores low because it lacks query terms. Debug: check whether the gold passage was in the candidate set and where it ranked.
  • Stale context. A superseded document is included because selection ignores version. Debug: compare document timestamps with the answer's claims.

In production

Log tokens by category and the excluded count per request, and alert when exclusion rises. Re-tune the allocation when the prompt or retrieval changes.

Common misconceptions

People think: A larger context window removes the need to select.

Actually: Selection still decides what the model attends to; larger windows add cost and dilution.

People think: Selection is the same as summarisation.

Actually: Selection chooses among existing material; summarisation rewrites it and can lose the detail you needed.

Interview angle

Frame selection as an allocation with a hard budget, then describe the ranking and the reserve for output. Strong answers separate mandatory from scored content and log exclusions. Weak answers say they put the top-k results in the prompt and never check what was displaced.

Sources