Why it matters
A language model can only answer from what reaches its context, so the retrieval stage decides the ceiling on answer quality. Information retrieval supplies the measurement discipline that keeps that stage honest: what counts as relevant, how much of the relevant set was found, and whether the ordering puts the best evidence first. Without those definitions, a team tunes similarity thresholds by feel and cannot tell a better index from a luckier query.
Mental model
Retrieval is a court filing system: the query is the request, the index is the registry, and relevance is whether a document answers the request rather than merely mentions it. The metaphor breaks because a registry clerk either finds a file or does not, while ranked retrieval always returns something, so the hard part is judging the near misses it hands back.
How it works
- 1Write the information need as a question plus the evidence that would settle it, so relevance has a definition before any index exists.
- 2Decide the retrieval unit, whether passage, section or record, because the unit fixes what a relevant result even means.
- 3Build a judged set of query-document pairs, sampling head queries, tail queries and queries with no answer.
- 4Measure recall at the depth you actually pass to the generator, then precision and nDCG within that depth.
- 5Separate matching failures from ranking failures by checking whether the correct document appears anywhere in the candidate list.
- 6Re-run the judged set whenever the tokenizer, embedding model or chunker changes, and keep the numbers beside the index version.
Step 1 of 6
Write the information need as a question plus the evidence that would settle it, so relevance has a definition before any index exists.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write the information need as a question plus the evidence that would settle it, so relev… 2. Decide the retrieval unit, whether passage, section or record, because the unit fixes wha… 3. Build a judged set of query-document pairs, sampling head queries, tail queries and queri… 4. Measure recall at the depth you actually pass to the generator, then precision and nDCG w… 5. Separate matching failures from ranking failures by checking whether the correct document… 6. Re-run the judged set whenever the tokenizer, embedding model or chunker changes, and kee…
- Write the information need as a question plus the evidence that would settle it, so relevance has a definition before any index exists.
- Decide the retrieval unit, whether passage, section or record, because the unit fixes what a relevant result even means.
- Build a judged set of query-document pairs, sampling head queries, tail queries and queries with no answer.
- Measure recall at the depth you actually pass to the generator, then precision and nDCG within that depth.
- Separate matching failures from ranking failures by checking whether the correct document appears anywhere in the candidate list.
- Re-run the judged set whenever the tokenizer, embedding model or chunker changes, and keep the numbers beside the index version.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Recall-first retrieval | Feeding the generator enough evidence | More context tokens and more distractors | The answer sits in one short passage and noise drowns it |
| Precision-first retrieval | Short context, cheap generation | Misses evidence phrased differently from the query | Users ask paraphrased or vocabulary-mismatched questions |
| Single ranked list | Simple evaluation and caching | One query formulation decides everything | The need is ambiguous or multi-part |
| Learned ranking | Handles paraphrase and intent | Judged data and retraining cost | Traffic is too sparse to label |
Good for
Feeding the generator enough evidence
Cost
More context tokens and more distractors
Fails when
The answer sits in one short passage and noise drowns it
- The answer sits in one short passage and noise drowns it
- Users ask paraphrased or vocabulary-mismatched questions
- The need is ambiguous or multi-part
- Traffic is too sparse to label
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Recall-first retrieval is good for Feeding the generator enough evidence, costs More context tokens and more distractors, and fails when The answer sits in one short passage and noise drowns it. Precision-first retrieval is good for Short context, cheap generation, costs Misses evidence phrased differently from the query, and fails when Users ask paraphrased or vocabulary-mismatched questions. Single ranked list is good for Simple evaluation and caching, costs One query formulation decides everything, and fails when The need is ambiguous or multi-part. Learned ranking is good for Handles paraphrase and intent, costs Judged data and retraining cost, and fails when Traffic is too sparse to label
Failure modes
- Unjudged relevant documents. Recall looks low because the labelled set omitted valid answers, so the team chases a phantom regression. Debug: pool the top results from every system under comparison and judge the union before comparing scores.
- Metric chosen for the wrong depth. Recall@100 is healthy while recall@5, the depth actually prompted, is poor. Debug: recompute every metric at the k used in production.
- Query-document asymmetry. Queries are short keyword strings and documents are long prose, so lexical overlap dominates the ranking. Debug: inspect the token overlap distribution for known-relevant pairs.
In production
Keep a frozen judged set with its index version and report recall at the production depth on every retrieval change. Alert when the zero-result rate or the mean rank of the known-good document moves.
Common misconceptions
People think: Retrieval is a solved problem because web search works.
Actually: Web search has billions of queries, click feedback and a link graph; a private corpus has none of those, so its relevance signals must be built.
People think: A relevant document is one that contains the query terms.
Actually: Term containment is a cheap proxy; relevance is whether the passage supplies the evidence the question needs.
Interview angle
Interviewers probe whether you can define relevance for a specific corpus and then measure it. Name the retrieval unit, the judged set and recall at the depth that reaches the model. Candidates who recite precision and recall formulas without tying them to a context budget or to real query shapes usually cannot explain why their retrieval regressed.
Sources
- Manning, Raghavan & Schütze, Introduction to Information Retrieval — the standard treatment of relevance, ranking and evaluation.
- Thakur et al., BEIR — why retrieval quality must be measured across tasks rather than one benchmark.
- Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond — the probabilistic framing of relevance and term weighting.