Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 15 of 28

Context metrics

0 of 28 complete

Why it matters

Teams improve retrieval and assume the answer improves with it, but the model may ignore the retrieved text, use a different chunk or answer from parametric memory. Context metrics close that gap by measuring the relationship between what was supplied and what was used. They also expose the cost side: every extra token raises latency and spend, and past a point it lowers accuracy by burying the evidence that mattered.

Mental model

It is a desk covered in papers where the one you need may be under the pile. The metaphor breaks because the model's pile has no physical order you can rearrange, only a position in a sequence that changes what gets attended to.

How it works

Mechanism, step by stepInteractive
  1. 1Log which retrieved chunks were included, their positions and their token counts.
  2. 2Score citation overlap by checking which chunks the answer's claims actually rely on.
  3. 3Run an ablation by removing a chunk and seeing whether the answer changes.
  4. 4Place the strongest evidence at the start and end, then compare accuracy with the middle position.
  5. 5Measure accuracy against context length on your own tasks rather than a synthetic needle test.
  6. 6Report context tokens per request alongside quality, since both move together.

Step 1 of 6

Log which retrieved chunks were included, their positions and their token counts.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Log which retrieved chunks were included, their positions and their token counts. 2. Score citation overlap by checking which chunks the answer's claims actually rely on. 3. Run an ablation by removing a chunk and seeing whether the answer changes. 4. Place the strongest evidence at the start and end, then compare accuracy with the middle … 5. Measure accuracy against context length on your own tasks rather than a synthetic needle … 6. Report context tokens per request alongside quality, since both move together.

Adding window costs the middleInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Share of answers that use the supplied evidence, against where the evidence sits in the window, at three window sizes. Attention is read at both ends and least reliable in the middle; every doubling of the advertised window lowers the middle faster than the ends. The U-shape is a stated model; the position effect is the claim.

What to notice. In an 8k window the evidence is used by 92% of answers at the start and 62% in the middle. At 128k the ends still hold at 84% while the middle falls to 42%, so the gap between edge and middle widens from 30 to 42 points. A larger window raises what can be included; it does not raise what gets used, and the loss lands on the evidence placed in the middle.

Text version

Three U-shaped curves of evidence use against its relative position in the window, from start to end. In the 8k window use is 92% at the start, 62% in the middle and 92% at the end. At 32k the middle is 52% and the ends 88%; at 128k the middle is 42% and the ends 84%. A vertical line marks the middle of the window and a marker on the 128k curve reports 42% there. A position slider reads all three curves at any point; a panel reports the middle value for each window size.

  1. Log which retrieved chunks were included, their positions and their token counts.
  2. Score citation overlap by checking which chunks the answer's claims actually rely on.
  3. Run an ablation by removing a chunk and seeing whether the answer changes.
  4. Place the strongest evidence at the start and end, then compare accuracy with the middle position.
  5. Measure accuracy against context length on your own tasks rather than a synthetic needle test.
  6. Report context tokens per request alongside quality, since both move together.

Trade-offs

ChoiceGood forCostFails when
Short curated contextHigher precision, lower costMisses recallAnswer needs several sources
Long context with everythingMaximum recallCost, distraction, middle lossRetriever is noisy
Rerank to a fixed budgetBalancedReranker errors, latencyRelevant chunk scores low
Position-aware orderingUses known attention effectsFragile to model changeContext is short enough to fit anywhere
Trade-off mapInteractive

Good for

Higher precision, lower cost

Cost

Misses recall

Fails when

Answer needs several sources

  1. Answer needs several sources
  2. Retriever is noisy
  3. Relevant chunk scores low
  4. Context is short enough to fit anywhere

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Short curated context is good for Higher precision, lower cost, costs Misses recall, and fails when Answer needs several sources. Long context with everything is good for Maximum recall, costs Cost, distraction, middle loss, and fails when Retriever is noisy. Rerank to a fixed budget is good for Balanced, costs Reranker errors, latency, and fails when Relevant chunk scores low. Position-aware ordering is good for Uses known attention effects, costs Fragile to model change, and fails when Context is short enough to fit anywhere

Failure modes

  • Retrieved but ignored. The supporting chunk is present and the answer contradicts it. Debug: remove the chunk and check whether the answer changes; if not, it was never used.
  • Middle burial. Accuracy drops when the evidence sits in the middle of a long prompt. Debug: move the same chunk to the start and compare.
  • Context inflation. Every request carries the full document and quality falls as cost rises. Debug: plot accuracy and tokens against context size.

In production

Track context tokens, citation coverage and accuracy by position bucket. Cap context and rerank rather than appending more text when quality drops.

Common misconceptions

People think: A larger context window solves retrieval failures.

Actually: It raises the ceiling on what can be included, not on what gets used.

People think: If the answer is correct, the context was used.

Actually: The model may answer from parametric knowledge and cite nothing, which fails when the source changes.

Interview angle

Separate retrieval from utilisation and describe a citation or ablation measurement. A strong answer knows the position effect and mentions token cost. Equating a bigger window with better answers, with no check on whether evidence changed the output, is the weak response.

Sources