Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 16 of 28

Ranking metrics

0 of 28 complete

Why it matters

Retrieval returns a ranked list, and the downstream step consumes only the first few items. A system that finds every relevant document but ranks them last is worse than one that finds fewer and puts them first. Ranking metrics make that distinction measurable, but only if the cutoff matches what the product actually reads and the relevance judgements are dense enough to trust.

Mental model

It is a queue at a door: who gets in and in what order. The metaphor breaks because the queue is judged against a list of who should have been inside, and that list is itself uncertain, so the metric inherits the judgement noise.

How it works

The winner depends on where you cutInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Precision against recall for two systems at twenty cut-offs, one per retrieved rank from k = 1 to k = 20, computed from two fixed ranked lists drawn from eight relevant documents. The curves cross, so the cut-off decides the ranking.

What to notice. At k = 5 the reranked list holds precision 60% at recall 37.5%, while the wide list holds 40% at 25%. At k = 20 the order reverses: 25% at 62.5% against 35% at 87.5%. Neither system is better in general; naming the cut-off before the comparison is what makes the number mean anything.

Text version

Precision from 0 to 100% against recall. Two stepped lines join twenty cut-off points, one per retrieved document. The reranked curve starts at 100% precision and falls to 25% at 62.5% recall; the wide curve falls faster early and ends higher, at 35% precision and 87.5% recall. The reranked list leads at k = 2, 4, 5 and 7; the two are level at k = 1, 3, 6, 8, 9, 12 and 13; and the wide list leads at k = 10, 11 and from k = 14 onward. Five buttons select a cut-off; the selected points on both curves are labelled with k and a panel reports precision and recall for each list and which one leads at that cut-off.

  1. Define relevance grades and how many results the downstream step actually consumes.
  2. Build judgements for a query sample with at least two assessors on a subset.
  3. Compute a metric that matches the consumer, such as MRR for a first hit or NDCG at the display cutoff.
  4. Report per-query distributions and the worst slice rather than only the mean.
  5. Fix the cutoff across compared systems and state it with the result.
  6. Re-judge a sample when the corpus or query mix changes.

Trade-offs

ChoiceGood forCostFails when
Precision at kSimple and interpretableIgnores order within kRanking position matters
Recall at kMeasures coverageIgnores how good the top results areUser reads only the first few
NDCGGraded relevance and positionNeeds graded judgementsRelevance is binary and shallow
MRRFocus on the first hitIgnores the rest of the listSeveral results are consumed
Trade-off mapInteractive

Good for

Simple and interpretable

Cost

Ignores order within k

Fails when

Ranking position matters

  1. Ranking position matters
  2. User reads only the first few
  3. Relevance is binary and shallow
  4. Several results are consumed

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Precision at k is good for Simple and interpretable, costs Ignores order within k, and fails when Ranking position matters. Recall at k is good for Measures coverage, costs Ignores how good the top results are, and fails when User reads only the first few. NDCG is good for Graded relevance and position, costs Needs graded judgements, and fails when Relevance is binary and shallow. MRR is good for Focus on the first hit, costs Ignores the rest of the list, and fails when Several results are consumed

Failure modes

  • Cutoff mismatch. A system wins at fifty results and loses at five where it is used. Debug: recompute both systems at the production cutoff.
  • Judge sparsity. Unjudged documents count as irrelevant and scores fall arbitrarily. Debug: report the judged fraction per query.
  • Metric gaming. Stuffing partially relevant items raises recall and hurts the user. Debug: pair recall with a precision or NDCG measure.

In production

Track the metric at the production cutoff per query slice. Re-judge a sample each quarter and monitor the unjudged rate.

Common misconceptions

People think: Recall is always the metric that matters.

Actually: It matters when coverage is the goal, while a list consumed top-down is better measured by NDCG or MRR.

People think: Ranking metrics measure answer quality.

Actually: They measure the order of candidates, and generation can still fail with perfect ranking.

Interview angle

Choose the metric from how many results are consumed and fix the cutoff. A strong answer reports per-query tails and pairs precision with recall. Quoting recall at an arbitrary k, with no view of the downstream step, is the weak answer.

Sources