Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 17 of 28

RAG metrics

0 of 28 complete

Why it matters

A RAG system fails at one of two handovers: the retriever supplies the wrong context, or the generator ignores the right context. An end-to-end score cannot say which happened, so a team may rewrite prompts when the index is the problem. Scoring retrieval and generation separately makes the failure local, and the ablation with retrieval removed shows how much the answer actually depends on the documents.

Mental model

It is a relay race where the baton can drop at either handover. The metaphor breaks because both runners can be fine while the race is run in the wrong direction, and only a separate check of the route catches that.

How it works

The average is nobody's itemInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. The per-item change in score when the retrieved context is removed, in two-point bins. Most items barely move and a block collapses; the mean change is a blend of the two, not a property of a typical item. The mixture is stated in the footnote and every point is generated.

What to notice. Removing the context costs 15.1 points on average, which reads as a retriever that matters everywhere. It does not: 35% of items move by less than five points in either direction and 37% lose more than twenty. The mean is carried by the collapsing block, and the unchanged block is where an end-to-end score cannot tell a working retriever from a decorative one.

Text version

Per-item score change from minus 74 to plus 18 points when the retrieved context is removed, in two-point bins. The distribution has a tall peak at zero, where 35% of items sit within plus or minus five points, a low valley near minus 20, a block near minus 32 and a smaller one near minus 58. A rule at minus 20 marks the cumulative cut, and the marker at the minus 32 block reports that 37% of items lie below the cut. Three steps introduce the average, the unchanged peak and the collapsing tail; a panel reports the share within five points, the mean change of minus 15.1 points and the share losing more than twenty.

  1. Log the retrieved chunks, their scores and the final answer for every evaluation item.
  2. Score context relevance against the question with graded judgements.
  3. Score groundedness by checking each answer claim against the retrieved text.
  4. Score answer relevance separately so a grounded but off-topic answer is visible.
  5. Run an ablation with retrieval removed to measure how much the answer depends on context.
  6. Report the three scores per item and inspect cases where they disagree.

Trade-offs

ChoiceGood forCostFails when
Reference-based answer metricsComparable and stableNeeds references, misses novel correct answersAnswers are open-ended
Reference-free groundednessWorks without labelsJudge cost and biasClaims need external verification
Retrieval-only metricsCheap and fastSay nothing about the answerGeneration is the weak stage
End-to-end human ratingRealisticCannot localise the failureYou need to fix the pipeline
Trade-off mapInteractive

Good for

Comparable and stable

Cost

Needs references, misses novel correct answers

Fails when

Answers are open-ended

  1. Answers are open-ended
  2. Claims need external verification
  3. Generation is the weak stage
  4. You need to fix the pipeline

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Reference-based answer metrics is good for Comparable and stable, costs Needs references, misses novel correct answers, and fails when Answers are open-ended. Reference-free groundedness is good for Works without labels, costs Judge cost and bias, and fails when Claims need external verification. Retrieval-only metrics is good for Cheap and fast, costs Say nothing about the answer, and fails when Generation is the weak stage. End-to-end human rating is good for Realistic, costs Cannot localise the failure, and fails when You need to fix the pipeline

Failure modes

  • Grounded but irrelevant. The answer faithfully summarises chunks that do not address the question. Debug: compare context relevance and groundedness scores for the same item.
  • Parametric answer. The answer is correct without using retrieval, hiding a broken retriever. Debug: rerun with an empty context and see whether the answer changes.
  • Chunk-boundary loss. The needed sentence is split across chunks and the answer invents the join. Debug: inspect the retrieved chunk boundaries around the cited text.

In production

Track retrieval and generation metrics as separate dashboards. Alert when groundedness falls while retrieval recall holds, which points at the prompt rather than the index.

Common misconceptions

People think: A good end-to-end score means both stages work.

Actually: A strong generator can mask a broken retriever, and a strong retriever can mask a careless generator.

People think: Groundedness and correctness are the same.

Actually: An answer can be grounded in wrong context, which makes it faithful and false.

Interview angle

Separate retrieval from generation, name a metric for each handover and describe the empty-context ablation. A strong answer explains how a correct answer can still indicate a broken retriever. One end-to-end score, with no localisation, is the weak response.

Sources