Why it matters
"The answers got worse this week" is not actionable. Something in ingestion, chunking, the embedding model, the index, the reranker, the prompt or the generator changed, and the only way to find out which is to have measured the stages independently before the regression. End-to-end quality scores are necessary but useless for localisation.
RAG evaluation is also how you justify architecture. A reranker, a query-rewriting step, a second retriever and a bigger generator all claim to improve quality; without a fixed labelled set, each addition is argued from a handful of anecdotes and the system accumulates complexity that nobody can attribute or remove.
Mental model
Evaluate the pipeline the way you would debug a factory: measure each station's output, not just the finished product. Incoming documents, retrieved candidates, assembled context, generated answer — each has a quality that can be checked before the next station sees it.
The metaphor breaks because the stations are not independent. Retrieval can only be scored well if you know what should have been retrieved, which requires labels on real questions, and those labels drift as the corpus changes. Evaluation is therefore a maintained dataset, not a one-off script.
How it works
- 1Define success in the product's terms. "The user gets a correct answer with valid citations, or a correct refusal" is a task definition. "High faithfulness score" is not.
- 2Build the golden set. Sample real queries, ideally including the long tail: short queries, identifier queries, multi-constraint questions, questions with no answer in the corpus. For each, label the supporting document or passage and, where feasible, a reference answer and the set of acceptable abstentions.
- 3Measure retrieval first. With evidence labels you can compute recall@k — the fraction of questions whose supporting passage appears in the top $k$ — and a ranking metric such as nDCG@k or MRR for ordering quality. Retrieval metrics are cheap, deterministic and the ceiling for everything downstream.
- 4Measure generation against the retrieved context. Decompose the answer into claims. For each claim, check whether the supplied context supports it (faithfulness), whether it addresses the question (relevance), and whether the answer covers the evidence that mattered (completeness). Check refusal behaviour on unanswerable questions.
- 5Check citations separately. Quote existence plus claim support, as described in citations. A grounded answer with fabricated references is not grounded.
- 6Slice every metric. By query type, document type, tenant, language and recency. Aggregate scores hide the slice that regressed.
Step 1 of 6
Define success in the product's terms. "The user gets a correct answer with valid citations, or a correct refusal" is a task definition. "High faithfulness score" is not.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Define success in the product's terms. "The user gets a correct answer with valid citatio… 2. Build the golden set. Sample real queries, ideally including the long tail: short queries… 3. Measure retrieval first. With evidence labels you can compute recall@k — the fraction of … 4. Measure generation against the retrieved context. Decompose the answer into claims. For e… 5. Check citations separately. Quote existence plus claim support, as described in citations… 6. Slice every metric. By query type, document type, tenant, language and recency. Aggregate…
- Define success in the product's terms. "The user gets a correct answer with valid citations, or a correct refusal" is a task definition. "High faithfulness score" is not.
- Build the golden set. Sample real queries, ideally including the long tail: short queries, identifier queries, multi-constraint questions, questions with no answer in the corpus. For each, label the supporting document or passage and, where feasible, a reference answer and the set of acceptable abstentions.
- Measure retrieval first. With evidence labels you can compute recall@k — the fraction of questions whose supporting passage appears in the top — and a ranking metric such as nDCG@k or MRR for ordering quality. Retrieval metrics are cheap, deterministic and the ceiling for everything downstream.
- Measure generation against the retrieved context. Decompose the answer into claims. For each claim, check whether the supplied context supports it (faithfulness), whether it addresses the question (relevance), and whether the answer covers the evidence that mattered (completeness). Check refusal behaviour on unanswerable questions.
- Check citations separately. Quote existence plus claim support, as described in citations. A grounded answer with fabricated references is not grounded.
- Slice every metric. By query type, document type, tenant, language and recency. Aggregate scores hide the slice that regressed.
- Automate and gate. Run the suite on every prompt, model or index change; block the change if the slices that matter move down beyond a threshold you defined in advance.
- Retrieval metrics
- recall@k, nDCG@k, MRR
- Answer metrics
- faithfulness, relevance, completeness
- Refusal metric
- correct abstention rate
- Needs humans
- Judge calibration and edge cases
- Runs in CI
- Fixed suite on every change
The two core formulas
Retrieval recall at , over a set of questions with labelled evidence :
Faithfulness, computed over the claims in an answer:
Both are only as good as their labels. A judge that is generous about "supported" inflates faithfulness; evidence labels that name the whole document rather than the passage inflate recall.
A minimal harness
def evaluate(pipeline, cases) -> dict:
rows = []
for case in cases:
trace = pipeline.run_with_trace(case.question)
retrieved = trace.candidates[:10]
recall = any(span.overlaps(doc) for doc in retrieved for span in case.evidence)
claims = decompose(trace.answer)
supported = [c for c in claims if judge.supports(c, trace.context)]
cited = [c for c in claims if c.cited]
# An abstention needs no citations; any other answer needs at least one,
# and every cited claim must resolve to a supporting span.
citations_ok = (trace.answer.is_abstention and not claims) or (
bool(cited) and all(check_citation(c, trace) for c in cited)
)
rows.append({
"id": case.id,
"recall@10": recall,
"faithfulness": len(supported) / max(len(claims), 1),
"citations_ok": citations_ok,
"refusal_correct": case.unanswerable and trace.answer.is_abstention,
"slice": case.slice,
})
return summarise(rows, group_by="slice")Keep the raw rows. Means hide the interesting cases, and every incident review starts by asking which questions were already failing before the change.
Trade-offs
| Method | Good for | Cost | Fails when |
|---|---|---|---|
| Retrieval metrics on labelled evidence | Localising regressions, cheap CI gate | Annotation effort, label drift | Evidence labels are document-level only |
| LLM judge on claims | Faithfulness and relevance at scale | Per-request cost, judge bias | The judge shares blind spots with the generator |
| NLI entailment model | Fast, cheap support checks | Misses implicit support and multi-hop reasoning | Claims require arithmetic or aggregation |
| Human review | Calibration, edge cases, high-stakes slices | Slow and expensive | The sample is too small to detect regressions |
| Synthetic questions | Cold start before real traffic exists | Distribution differs from real queries | Questions are generated from the chunks they should retrieve |
Good for
Localising regressions, cheap CI gate
Cost
Annotation effort, label drift
Fails when
Evidence labels are document-level only
- Evidence labels are document-level only
- The judge shares blind spots with the generator
- Claims require arithmetic or aggregation
- The sample is too small to detect regressions
- Questions are generated from the chunks they should retrieve
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Retrieval metrics on labelled evidence is good for Localising regressions, cheap CI gate, costs Annotation effort, label drift, and fails when Evidence labels are document-level only. LLM judge on claims is good for Faithfulness and relevance at scale, costs Per-request cost, judge bias, and fails when The judge shares blind spots with the generator. NLI entailment model is good for Fast, cheap support checks, costs Misses implicit support and multi-hop reasoning, and fails when Claims require arithmetic or aggregation. Human review is good for Calibration, edge cases, high-stakes slices, costs Slow and expensive, and fails when The sample is too small to detect regressions. Synthetic questions is good for Cold start before real traffic exists, costs Distribution differs from real queries, and fails when Questions are generated from the chunks they should retrieve
Failure modes
- Judge agrees with the generator. A judge from the same model family rates its own outputs generously, so faithfulness looks high while users complain. Debug: score a set of deliberately unsupported claims and check the judge rejects them; then compare judge verdicts against 50 human labels.
- Metric gaming. Longer answers contain more supported claims and score better on recall-like measures, so the system learns to pad. Debug: plot answer length against score and add a precision-style metric that penalises unsupported or irrelevant claims.
- Leakage into the dev set. Synthetic questions generated from a chunk are trivially retrieved by that chunk, inflating both retrieval and answer scores. Debug: review 20 synthetic questions by hand; if they reuse chunk phrasing verbatim, regenerate them from documents with the chunk hidden.
- No abstention cases. The suite contains only answerable questions, so a system that always answers looks perfect. Debug: add questions whose answers are absent from the corpus and score correct refusals explicitly.
In production
Judging every request is expensive, so production usually runs a cheap online proxy (abstain rate, citation verification failures, retrieval score distribution, user feedback) and a richer offline suite on a schedule and in CI. Version the eval set with the corpus: when documents are deleted, some labels become unanswerable and the suite must say so.
Track suite results per slice over time, not just the mean, and keep a small human-reviewed sample in the loop to catch judge drift. A judge model upgrade is a measurement change; re-baseline rather than comparing across versions.
Common misconceptions
People think: A single faithfulness score tells you the system is good.
Actually: Faithfulness only says the answer is supported by whatever was retrieved. A perfectly faithful answer to the wrong evidence still fails the user.
People think: If retrieval returns something plausible, retrieval is working.
Actually: Plausible is the failure mode. Retrieval quality means the supporting passage is present, which requires labels, not inspection.
Interview angle
The question is usually "how would you evaluate this RAG system", and interviewers push on two things: whether you separate retrieval from generation, and whether your metrics can attribute a regression. A strong answer names recall@k with labelled evidence, claim-level faithfulness, correct abstention, and slice-level reporting, then explains how the suite gates a change. See RAG interview questions and why evaluation.
Sources
- Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation — reference-free metrics for retrieval and generation quality.
- Saad-Falcon et al., ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems — judge-based evaluation with calibration against human annotations.
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — documented biases in judge models and agreement with human preference.
- Thakur et al., BEIR — retrieval evaluation conventions across heterogeneous datasets.