Why it matters
Retrieval returns a ranked list, and the downstream step consumes only the first few items. A system that finds every relevant document but ranks them last is worse than one that finds fewer and puts them first. Ranking metrics make that distinction measurable, but only if the cutoff matches what the product actually reads and the relevance judgements are dense enough to trust.
Mental model
It is a queue at a door: who gets in and in what order. The metaphor breaks because the queue is judged against a list of who should have been inside, and that list is itself uncertain, so the metric inherits the judgement noise.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Precision against recall for two systems at twenty cut-offs, one per retrieved rank from k = 1 to k = 20, computed from two fixed ranked lists drawn from eight relevant documents. The curves cross, so the cut-off decides the ranking.
What to notice. At k = 5 the reranked list holds precision 60% at recall 37.5%, while the wide list holds 40% at 25%. At k = 20 the order reverses: 25% at 62.5% against 35% at 87.5%. Neither system is better in general; naming the cut-off before the comparison is what makes the number mean anything.
Text version
Precision from 0 to 100% against recall. Two stepped lines join twenty cut-off points, one per retrieved document. The reranked curve starts at 100% precision and falls to 25% at 62.5% recall; the wide curve falls faster early and ends higher, at 35% precision and 87.5% recall. The reranked list leads at k = 2, 4, 5 and 7; the two are level at k = 1, 3, 6, 8, 9, 12 and 13; and the wide list leads at k = 10, 11 and from k = 14 onward. Five buttons select a cut-off; the selected points on both curves are labelled with k and a panel reports precision and recall for each list and which one leads at that cut-off.
- Define relevance grades and how many results the downstream step actually consumes.
- Build judgements for a query sample with at least two assessors on a subset.
- Compute a metric that matches the consumer, such as MRR for a first hit or NDCG at the display cutoff.
- Report per-query distributions and the worst slice rather than only the mean.
- Fix the cutoff across compared systems and state it with the result.
- Re-judge a sample when the corpus or query mix changes.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Precision at k | Simple and interpretable | Ignores order within k | Ranking position matters |
| Recall at k | Measures coverage | Ignores how good the top results are | User reads only the first few |
| NDCG | Graded relevance and position | Needs graded judgements | Relevance is binary and shallow |
| MRR | Focus on the first hit | Ignores the rest of the list | Several results are consumed |
Good for
Simple and interpretable
Cost
Ignores order within k
Fails when
Ranking position matters
- Ranking position matters
- User reads only the first few
- Relevance is binary and shallow
- Several results are consumed
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Precision at k is good for Simple and interpretable, costs Ignores order within k, and fails when Ranking position matters. Recall at k is good for Measures coverage, costs Ignores how good the top results are, and fails when User reads only the first few. NDCG is good for Graded relevance and position, costs Needs graded judgements, and fails when Relevance is binary and shallow. MRR is good for Focus on the first hit, costs Ignores the rest of the list, and fails when Several results are consumed
Failure modes
- Cutoff mismatch. A system wins at fifty results and loses at five where it is used. Debug: recompute both systems at the production cutoff.
- Judge sparsity. Unjudged documents count as irrelevant and scores fall arbitrarily. Debug: report the judged fraction per query.
- Metric gaming. Stuffing partially relevant items raises recall and hurts the user. Debug: pair recall with a precision or NDCG measure.
In production
Track the metric at the production cutoff per query slice. Re-judge a sample each quarter and monitor the unjudged rate.
Common misconceptions
People think: Recall is always the metric that matters.
Actually: It matters when coverage is the goal, while a list consumed top-down is better measured by NDCG or MRR.
People think: Ranking metrics measure answer quality.
Actually: They measure the order of candidates, and generation can still fail with perfect ranking.
Interview angle
Choose the metric from how many results are consumed and fix the cutoff. A strong answer reports per-query tails and pairs precision with recall. Quoting recall at an arbitrary k, with no view of the downstream step, is the weak answer.
Sources
- Järvelin and Kekäläinen, Cumulated Gain-based Evaluation of IR Techniques — the NDCG formulation.
- trec_eval — the reference implementation of standard IR measures.
- pytrec_eval — a Python binding used to compute these metrics consistently.