Why it matters
Every retrieved passage competes for the model's attention and for the context window. Set k too low and the answer is never in the prompt; set it too high and the correct passage is buried among plausible neighbours, which is worse than a short clean context. Top-k is where retrieval recall and generation quality meet, and it has to be chosen against both.
Mental model
Top-k is the number of search results you hand a researcher before asking for a summary. Too few and the fact is missing; too many and the relevant page is lost in the stack. The metaphor breaks because the researcher reads everything, while a model's use of a passage depends on where in the context it sits.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Twenty synthetic passages ranked by euclidean distance to one query; four are marked relevant. k is the cut-off, so it sets both how much of the evidence is retrieved and how much noise travels with it.
What to notice. At k = 1 one of four relevant passages is retrieved: recall 25%, precision 100%. At k = 5 two are retrieved (50% recall) and 40% of the retrieved passages are relevant. At k = 20 all four are retrieved and one result in five is relevant. The figure measures one fixed ranking, so it shows the cost of depth, not the benefit of a reranker.
Text version
Twenty synthetic passages arranged around a query, four marked relevant, with a slider from k = 1 to 20. The default k = 5 rings and joins the five nearest passages: refund policy at rank one and returns at rank four are relevant, so recall is 50% and precision 40%. Dragging to k = 1 leaves only refund policy, for 25% recall at 100% precision; dragging to k = 20 includes money back at rank nine and warranty at rank fourteen, for 100% recall at 20% precision. Two bars under the rank panel track recall and precision as k changes.
- Measure recall at several depths on a labelled set to find where the curve flattens.
- Compute the token cost of k passages at your average chunk size and compare with the context budget.
- Evaluate answer quality, not just recall, at each depth, holding the prompt and model fixed.
- Add a reranker and reduce k afterwards, since reranking makes a smaller set viable.
- Introduce an adaptive rule: expand k when the score gap between neighbours is small, shrink it when the top hit is far ahead.
- Place the strongest passages at the start and end of the context rather than in the middle.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Small k | Short prompt, low cost, less noise | Misses evidence when recall is imperfect | First-stage recall at that depth is low |
| Large k | High evidence coverage | Cost, latency and attention dilution | Distractors resemble the answer |
| Fixed k | Predictable cost and simple caching | Over-serves easy queries and under-serves hard ones | Query difficulty varies widely |
| Adaptive k | Matches depth to difficulty | Needs a reliable score signal and more code | Similarity scores are uncalibrated |
Good for
Short prompt, low cost, less noise
Cost
Misses evidence when recall is imperfect
Fails when
First-stage recall at that depth is low
- First-stage recall at that depth is low
- Distractors resemble the answer
- Query difficulty varies widely
- Similarity scores are uncalibrated
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Small k is good for Short prompt, low cost, less noise, costs Misses evidence when recall is imperfect, and fails when First-stage recall at that depth is low. Large k is good for High evidence coverage, costs Cost, latency and attention dilution, and fails when Distractors resemble the answer. Fixed k is good for Predictable cost and simple caching, costs Over-serves easy queries and under-serves hard ones, and fails when Query difficulty varies widely. Adaptive k is good for Matches depth to difficulty, costs Needs a reliable score signal and more code, and fails when Similarity scores are uncalibrated
Failure modes
- k chosen from recall alone. Recall is high and answers get worse because the prompt is full of near-misses. Debug: compare answer correctness at k and at half of k with the same reranker.
- Context truncation at the tail. The prompt builder cuts the last passages, so effective k is smaller than configured. Debug: log tokens before and after assembly and count passages that survived.
- Middle burial. The correct passage is retrieved but placed mid-context and ignored. Debug: move the gold passage to the front for failing queries and see whether the answer changes.
In production
Log effective k after truncation, prompt tokens and answer quality per depth. Alert when the configured k and the number of passages actually in the prompt diverge.
Common misconceptions
People think: More context is always better.
Actually: Beyond the point where recall flattens, extra passages add cost and distractors that reduce accuracy.
People think: Recall at k is the same as answer quality at k.
Actually: Retrieval can succeed while the generator ignores or misreads the passage it was given.
Interview angle
Treat k as a joint retrieval and context decision, cite the recall curve and the attention effect, and describe an adaptive rule. Keep recall and answer quality distinct. Stating a number for k without connecting it to chunk size, recall or the context budget is the answer that cannot be defended.
Sources
- Liu et al., Lost in the Middle — how position within a long context changes use of the evidence.
- Thakur et al., BEIR — recall at k as the retrieval measure that depth controls.
- Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey — context selection as a stage between retrieval and generation.