Why it matters
A RAG system fails at one of two handovers: the retriever supplies the wrong context, or the generator ignores the right context. An end-to-end score cannot say which happened, so a team may rewrite prompts when the index is the problem. Scoring retrieval and generation separately makes the failure local, and the ablation with retrieval removed shows how much the answer actually depends on the documents.
Mental model
It is a relay race where the baton can drop at either handover. The metaphor breaks because both runners can be fine while the race is run in the wrong direction, and only a separate check of the route catches that.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. The per-item change in score when the retrieved context is removed, in two-point bins. Most items barely move and a block collapses; the mean change is a blend of the two, not a property of a typical item. The mixture is stated in the footnote and every point is generated.
What to notice. Removing the context costs 15.1 points on average, which reads as a retriever that matters everywhere. It does not: 35% of items move by less than five points in either direction and 37% lose more than twenty. The mean is carried by the collapsing block, and the unchanged block is where an end-to-end score cannot tell a working retriever from a decorative one.
Text version
Per-item score change from minus 74 to plus 18 points when the retrieved context is removed, in two-point bins. The distribution has a tall peak at zero, where 35% of items sit within plus or minus five points, a low valley near minus 20, a block near minus 32 and a smaller one near minus 58. A rule at minus 20 marks the cumulative cut, and the marker at the minus 32 block reports that 37% of items lie below the cut. Three steps introduce the average, the unchanged peak and the collapsing tail; a panel reports the share within five points, the mean change of minus 15.1 points and the share losing more than twenty.
- Log the retrieved chunks, their scores and the final answer for every evaluation item.
- Score context relevance against the question with graded judgements.
- Score groundedness by checking each answer claim against the retrieved text.
- Score answer relevance separately so a grounded but off-topic answer is visible.
- Run an ablation with retrieval removed to measure how much the answer depends on context.
- Report the three scores per item and inspect cases where they disagree.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Reference-based answer metrics | Comparable and stable | Needs references, misses novel correct answers | Answers are open-ended |
| Reference-free groundedness | Works without labels | Judge cost and bias | Claims need external verification |
| Retrieval-only metrics | Cheap and fast | Say nothing about the answer | Generation is the weak stage |
| End-to-end human rating | Realistic | Cannot localise the failure | You need to fix the pipeline |
Good for
Comparable and stable
Cost
Needs references, misses novel correct answers
Fails when
Answers are open-ended
- Answers are open-ended
- Claims need external verification
- Generation is the weak stage
- You need to fix the pipeline
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Reference-based answer metrics is good for Comparable and stable, costs Needs references, misses novel correct answers, and fails when Answers are open-ended. Reference-free groundedness is good for Works without labels, costs Judge cost and bias, and fails when Claims need external verification. Retrieval-only metrics is good for Cheap and fast, costs Say nothing about the answer, and fails when Generation is the weak stage. End-to-end human rating is good for Realistic, costs Cannot localise the failure, and fails when You need to fix the pipeline
Failure modes
- Grounded but irrelevant. The answer faithfully summarises chunks that do not address the question. Debug: compare context relevance and groundedness scores for the same item.
- Parametric answer. The answer is correct without using retrieval, hiding a broken retriever. Debug: rerun with an empty context and see whether the answer changes.
- Chunk-boundary loss. The needed sentence is split across chunks and the answer invents the join. Debug: inspect the retrieved chunk boundaries around the cited text.
In production
Track retrieval and generation metrics as separate dashboards. Alert when groundedness falls while retrieval recall holds, which points at the prompt rather than the index.
Common misconceptions
People think: A good end-to-end score means both stages work.
Actually: A strong generator can mask a broken retriever, and a strong retriever can mask a careless generator.
People think: Groundedness and correctness are the same.
Actually: An answer can be grounded in wrong context, which makes it faithful and false.
Interview angle
Separate retrieval from generation, name a metric for each handover and describe the empty-context ablation. A strong answer explains how a correct answer can still indicate a broken retriever. One end-to-end score, with no localisation, is the weak response.
Sources
- Es et al., RAGAS — reference-free metrics for retrieval-augmented generation.
- Saad-Falcon et al., ARES — automated evaluation of RAG with trained judges.
- Yang et al., CRAG — a benchmark covering retrieval and generation failures.