Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 33 of 57

Top-k

0 of 57 complete

Why it matters

Every retrieved passage competes for the model's attention and for the context window. Set k too low and the answer is never in the prompt; set it too high and the correct passage is buried among plausible neighbours, which is worse than a short clean context. Top-k is where retrieval recall and generation quality meet, and it has to be chosen against both.

Mental model

Top-k is the number of search results you hand a researcher before asking for a summary. Too few and the fact is missing; too many and the relevant page is lost in the stack. The metaphor breaks because the researcher reads everything, while a model's use of a passage depends on where in the context it sits.

How it works

Top-k · recall bought with precisionInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Twenty synthetic passages ranked by euclidean distance to one query; four are marked relevant. k is the cut-off, so it sets both how much of the evidence is retrieved and how much noise travels with it.

What to notice. At k = 1 one of four relevant passages is retrieved: recall 25%, precision 100%. At k = 5 two are retrieved (50% recall) and 40% of the retrieved passages are relevant. At k = 20 all four are retrieved and one result in five is relevant. The figure measures one fixed ranking, so it shows the cost of depth, not the benefit of a reranker.

Text version

Twenty synthetic passages arranged around a query, four marked relevant, with a slider from k = 1 to 20. The default k = 5 rings and joins the five nearest passages: refund policy at rank one and returns at rank four are relevant, so recall is 50% and precision 40%. Dragging to k = 1 leaves only refund policy, for 25% recall at 100% precision; dragging to k = 20 includes money back at rank nine and warranty at rank fourteen, for 100% recall at 20% precision. Two bars under the rank panel track recall and precision as k changes.

  1. Measure recall at several depths on a labelled set to find where the curve flattens.
  2. Compute the token cost of k passages at your average chunk size and compare with the context budget.
  3. Evaluate answer quality, not just recall, at each depth, holding the prompt and model fixed.
  4. Add a reranker and reduce k afterwards, since reranking makes a smaller set viable.
  5. Introduce an adaptive rule: expand k when the score gap between neighbours is small, shrink it when the top hit is far ahead.
  6. Place the strongest passages at the start and end of the context rather than in the middle.

Trade-offs

ChoiceGood forCostFails when
Small kShort prompt, low cost, less noiseMisses evidence when recall is imperfectFirst-stage recall at that depth is low
Large kHigh evidence coverageCost, latency and attention dilutionDistractors resemble the answer
Fixed kPredictable cost and simple cachingOver-serves easy queries and under-serves hard onesQuery difficulty varies widely
Adaptive kMatches depth to difficultyNeeds a reliable score signal and more codeSimilarity scores are uncalibrated
Trade-off mapInteractive

Good for

Short prompt, low cost, less noise

Cost

Misses evidence when recall is imperfect

Fails when

First-stage recall at that depth is low

  1. First-stage recall at that depth is low
  2. Distractors resemble the answer
  3. Query difficulty varies widely
  4. Similarity scores are uncalibrated

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Small k is good for Short prompt, low cost, less noise, costs Misses evidence when recall is imperfect, and fails when First-stage recall at that depth is low. Large k is good for High evidence coverage, costs Cost, latency and attention dilution, and fails when Distractors resemble the answer. Fixed k is good for Predictable cost and simple caching, costs Over-serves easy queries and under-serves hard ones, and fails when Query difficulty varies widely. Adaptive k is good for Matches depth to difficulty, costs Needs a reliable score signal and more code, and fails when Similarity scores are uncalibrated

Failure modes

  • k chosen from recall alone. Recall is high and answers get worse because the prompt is full of near-misses. Debug: compare answer correctness at k and at half of k with the same reranker.
  • Context truncation at the tail. The prompt builder cuts the last passages, so effective k is smaller than configured. Debug: log tokens before and after assembly and count passages that survived.
  • Middle burial. The correct passage is retrieved but placed mid-context and ignored. Debug: move the gold passage to the front for failing queries and see whether the answer changes.

In production

Log effective k after truncation, prompt tokens and answer quality per depth. Alert when the configured k and the number of passages actually in the prompt diverge.

Common misconceptions

People think: More context is always better.

Actually: Beyond the point where recall flattens, extra passages add cost and distractors that reduce accuracy.

People think: Recall at k is the same as answer quality at k.

Actually: Retrieval can succeed while the generator ignores or misreads the passage it was given.

Interview angle

Treat k as a joint retrieval and context decision, cite the recall curve and the attention effect, and describe an adaptive rule. Keep recall and answer quality distinct. Stating a number for k without connecting it to chunk size, recall or the context budget is the answer that cannot be defended.

Sources