Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 1 of 57

IR

0 of 57 complete

Why it matters

A language model can only answer from what reaches its context, so the retrieval stage decides the ceiling on answer quality. Information retrieval supplies the measurement discipline that keeps that stage honest: what counts as relevant, how much of the relevant set was found, and whether the ordering puts the best evidence first. Without those definitions, a team tunes similarity thresholds by feel and cannot tell a better index from a luckier query.

Mental model

Retrieval is a court filing system: the query is the request, the index is the registry, and relevance is whether a document answers the request rather than merely mentions it. The metaphor breaks because a registry clerk either finds a file or does not, while ranked retrieval always returns something, so the hard part is judging the near misses it hands back.

How it works

Mechanism, step by stepInteractive
  1. 1Write the information need as a question plus the evidence that would settle it, so relevance has a definition before any index exists.
  2. 2Decide the retrieval unit, whether passage, section or record, because the unit fixes what a relevant result even means.
  3. 3Build a judged set of query-document pairs, sampling head queries, tail queries and queries with no answer.
  4. 4Measure recall at the depth you actually pass to the generator, then precision and nDCG within that depth.
  5. 5Separate matching failures from ranking failures by checking whether the correct document appears anywhere in the candidate list.
  6. 6Re-run the judged set whenever the tokenizer, embedding model or chunker changes, and keep the numbers beside the index version.

Step 1 of 6

Write the information need as a question plus the evidence that would settle it, so relevance has a definition before any index exists.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the information need as a question plus the evidence that would settle it, so relev… 2. Decide the retrieval unit, whether passage, section or record, because the unit fixes wha… 3. Build a judged set of query-document pairs, sampling head queries, tail queries and queri… 4. Measure recall at the depth you actually pass to the generator, then precision and nDCG w… 5. Separate matching failures from ranking failures by checking whether the correct document… 6. Re-run the judged set whenever the tokenizer, embedding model or chunker changes, and kee…

  1. Write the information need as a question plus the evidence that would settle it, so relevance has a definition before any index exists.
  2. Decide the retrieval unit, whether passage, section or record, because the unit fixes what a relevant result even means.
  3. Build a judged set of query-document pairs, sampling head queries, tail queries and queries with no answer.
  4. Measure recall at the depth you actually pass to the generator, then precision and nDCG within that depth.
  5. Separate matching failures from ranking failures by checking whether the correct document appears anywhere in the candidate list.
  6. Re-run the judged set whenever the tokenizer, embedding model or chunker changes, and keep the numbers beside the index version.

Trade-offs

ChoiceGood forCostFails when
Recall-first retrievalFeeding the generator enough evidenceMore context tokens and more distractorsThe answer sits in one short passage and noise drowns it
Precision-first retrievalShort context, cheap generationMisses evidence phrased differently from the queryUsers ask paraphrased or vocabulary-mismatched questions
Single ranked listSimple evaluation and cachingOne query formulation decides everythingThe need is ambiguous or multi-part
Learned rankingHandles paraphrase and intentJudged data and retraining costTraffic is too sparse to label
Trade-off mapInteractive

Good for

Feeding the generator enough evidence

Cost

More context tokens and more distractors

Fails when

The answer sits in one short passage and noise drowns it

  1. The answer sits in one short passage and noise drowns it
  2. Users ask paraphrased or vocabulary-mismatched questions
  3. The need is ambiguous or multi-part
  4. Traffic is too sparse to label

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Recall-first retrieval is good for Feeding the generator enough evidence, costs More context tokens and more distractors, and fails when The answer sits in one short passage and noise drowns it. Precision-first retrieval is good for Short context, cheap generation, costs Misses evidence phrased differently from the query, and fails when Users ask paraphrased or vocabulary-mismatched questions. Single ranked list is good for Simple evaluation and caching, costs One query formulation decides everything, and fails when The need is ambiguous or multi-part. Learned ranking is good for Handles paraphrase and intent, costs Judged data and retraining cost, and fails when Traffic is too sparse to label

Failure modes

  • Unjudged relevant documents. Recall looks low because the labelled set omitted valid answers, so the team chases a phantom regression. Debug: pool the top results from every system under comparison and judge the union before comparing scores.
  • Metric chosen for the wrong depth. Recall@100 is healthy while recall@5, the depth actually prompted, is poor. Debug: recompute every metric at the k used in production.
  • Query-document asymmetry. Queries are short keyword strings and documents are long prose, so lexical overlap dominates the ranking. Debug: inspect the token overlap distribution for known-relevant pairs.

In production

Keep a frozen judged set with its index version and report recall at the production depth on every retrieval change. Alert when the zero-result rate or the mean rank of the known-good document moves.

Common misconceptions

People think: Retrieval is a solved problem because web search works.

Actually: Web search has billions of queries, click feedback and a link graph; a private corpus has none of those, so its relevance signals must be built.

People think: A relevant document is one that contains the query terms.

Actually: Term containment is a cheap proxy; relevance is whether the passage supplies the evidence the question needs.

Interview angle

Interviewers probe whether you can define relevance for a specific corpus and then measure it. Name the retrieval unit, the judged set and recall at the depth that reaches the model. Candidates who recite precision and recall formulas without tying them to a context budget or to real query shapes usually cannot explain why their retrieval regressed.

Sources