Why it matters
Adding a retriever to a prompt does not automatically ground the answer. The model still has to treat the passages as the source of truth, prefer them over parametric memory, cite them and abstain when they are irrelevant. Each of those is a prompt-level behaviour that has to be specified and evaluated. The engineering problem is separating retrieval quality from prompt quality when the final answer is wrong.
Mental model
Retrieval as context is handing an expert a folder of documents before asking a question. The comparison breaks because the expert can tell a folder is irrelevant, while the model may answer from memory and cite the folder anyway. The citation looks like evidence of reading.
How it works
- 1Format each passage with an id, a source and a clear boundary, and state that passages are evidence rather than instructions.
- 2Instruct the model to answer only from the passages and to name the id supporting each claim.
- 3Define abstention behaviour for missing or contradictory evidence, with the exact wording to use.
- 4Place the most relevant passages near the question and keep ids stable across the prompt.
- 5Log retrieved ids, cited ids and the final answer together, so retrieval and generation failures are distinguishable.
- 6Evaluate with retrieval held fixed and with retrieval ablated, to see what the passages contribute.
Step 1 of 6
Format each passage with an id, a source and a clear boundary, and state that passages are evidence rather than instructions.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Format each passage with an id, a source and a clear boundary, and state that passages ar… 2. Instruct the model to answer only from the passages and to name the id supporting each cl… 3. Define abstention behaviour for missing or contradictory evidence, with the exact wording… 4. Place the most relevant passages near the question and keep ids stable across the prompt. 5. Log retrieved ids, cited ids and the final answer together, so retrieval and generation f… 6. Evaluate with retrieval held fixed and with retrieval ablated, to see what the passages c…
- Format each passage with an id, a source and a clear boundary, and state that passages are evidence rather than instructions.
- Instruct the model to answer only from the passages and to name the id supporting each claim.
- Define abstention behaviour for missing or contradictory evidence, with the exact wording to use.
- Place the most relevant passages near the question and keep ids stable across the prompt.
- Log retrieved ids, cited ids and the final answer together, so retrieval and generation failures are distinguishable.
- Evaluate with retrieval held fixed and with retrieval ablated, to see what the passages contribute.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Strict grounding | Auditable, fewer unsupported claims | More abstentions | The corpus has gaps the user expects filled |
| Loose grounding | Helpful answers from partial evidence | Unsupported claims mixed with cited ones | The domain is regulated |
| Many passages | Higher recall | Dilution and longer prompts | Only one passage matters |
| Few passages | Focused context | Misses multi-hop evidence | The question needs several sources |
Good for
Auditable, fewer unsupported claims
Cost
More abstentions
Fails when
The corpus has gaps the user expects filled
- The corpus has gaps the user expects filled
- The domain is regulated
- Only one passage matters
- The question needs several sources
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Strict grounding is good for Auditable, fewer unsupported claims, costs More abstentions, and fails when The corpus has gaps the user expects filled. Loose grounding is good for Helpful answers from partial evidence, costs Unsupported claims mixed with cited ones, and fails when The domain is regulated. Many passages is good for Higher recall, costs Dilution and longer prompts, and fails when Only one passage matters. Few passages is good for Focused context, costs Misses multi-hop evidence, and fails when The question needs several sources
Failure modes
- Citation without support. The answer cites a passage that does not contain the claim. Debug: check each cited id against the claim's entities in the passage text.
- Memory override. The model answers from training data when passages are thin. Debug: run the same question with retrieval ablated and compare the answers.
- Irrelevant context accepted. The retriever returns near-misses and the model answers anyway. Debug: add unanswerable questions and measure the abstention rate.
In production
Track retrieval hit rate, citation validity and abstention rate as separate metrics. Alert when a retrieval outage produces more answers rather than more abstentions.
Common misconceptions
People think: Retrieval eliminates the need for prompt rules.
Actually: The model still needs instructions about evidence, citation and abstention.
People think: A wrong answer means retrieval failed.
Actually: It may be a prompt that ignored good evidence; logging ids separates the two.
Interview angle
Explain that retrieval and prompting are two failure surfaces, then describe how you separate them with ids and ablation. Strong answers include citation checks and abstention. Weak answers add a retriever and assume the answer is now grounded.
Sources
- Lewis et al., Retrieval-Augmented Generation — conditioning generation on retrieved passages.
- Anthropic, Contextual Retrieval — preparing retrieved chunks so they remain interpretable in context.
- Liu et al., Lost in the Middle — why placement of retrieved evidence matters.