Why it matters
An eval is a proxy, and a proxy that disagrees with the people using the product is worse than no eval because it creates false confidence. The disagreement has a small number of causes: the labelled set does not resemble production, the rubric is ambiguous, the judge has a systematic bias, or the humans are reacting to something the metric does not measure, such as tone or latency. Each cause needs a different investigation, so the first step is to find examples rather than debate the score.
Mental model
It is a smoke alarm that stays silent during a fire: either the sensor is pointed elsewhere, the threshold is wrong, or it is measuring a different kind of smoke. The metaphor breaks because humans also disagree with each other, so there is no single ground truth to calibrate against.
How it works
- 1Collect ten cases where the eval scored well and a human rated the output poorly.
- 2Read the outputs and write what the human objected to, in their words.
- 3Check whether the objection is covered by the rubric; if not, the metric is measuring something else.
- 4Measure inter-rater agreement on a sample, since low agreement means the label is unstable.
- 5Compare the eval set's distribution with production traffic on length, topic and query type.
- 6Change one thing, either the rubric, the judge prompt or the sample, and re-measure agreement on the same cases.
Step 1 of 6
Collect ten cases where the eval scored well and a human rated the output poorly.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Collect ten cases where the eval scored well and a human rated the output poorly. 2. Read the outputs and write what the human objected to, in their words. 3. Check whether the objection is covered by the rubric; if not, the metric is measuring som… 4. Measure inter-rater agreement on a sample, since low agreement means the label is unstabl… 5. Compare the eval set's distribution with production traffic on length, topic and query ty… 6. Change one thing, either the rubric, the judge prompt or the sample, and re-measure agree…
- Collect ten cases where the eval scored well and a human rated the output poorly.
- Read the outputs and write what the human objected to, in their words.
- Check whether the objection is covered by the rubric; if not, the metric is measuring something else.
- Measure inter-rater agreement on a sample, since low agreement means the label is unstable.
- Compare the eval set's distribution with production traffic on length, topic and query type.
- Change one thing, either the rubric, the judge prompt or the sample, and re-measure agreement on the same cases.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Trust the humans | Catches what the metric misses | Human judgement is noisy and expensive | The complaint comes from one vocal user |
| Trust the eval | Consistent and cheap | Systematic blind spots persist | The rubric was written before the product changed |
| Add a new metric | Covers the missing dimension | More numbers to reconcile | Nobody agrees which metric decides |
| Re-weight the set | Reflects production | Old regressions become invisible | The tail is the part that matters |
Good for
Catches what the metric misses
Cost
Human judgement is noisy and expensive
Fails when
The complaint comes from one vocal user
- The complaint comes from one vocal user
- The rubric was written before the product changed
- Nobody agrees which metric decides
- The tail is the part that matters
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Trust the humans is good for Catches what the metric misses, costs Human judgement is noisy and expensive, and fails when The complaint comes from one vocal user. Trust the eval is good for Consistent and cheap, costs Systematic blind spots persist, and fails when The rubric was written before the product changed. Add a new metric is good for Covers the missing dimension, costs More numbers to reconcile, and fails when Nobody agrees which metric decides. Re-weight the set is good for Reflects production, costs Old regressions become invisible, and fails when The tail is the part that matters
Failure modes
- Rubric gap. Humans penalise tone and the metric only checks factuality. Debug: read ten disagreements and tag the objection against the rubric dimensions.
- Label noise. Two annotators disagree and the eval is chasing an unstable target. Debug: compute agreement between annotators on the same sample.
- Distribution shift. The eval set is drawn from launch traffic and production has moved on. Debug: compare query length and topic distributions between the set and last week's logs.
In production
Track eval scores and human review outcomes on the same dashboard, and alert when they diverge for a week. Keep a small freshly sampled set to detect drift.
Common misconceptions
People think: If the eval is well built, human disagreement means the humans are wrong.
Actually: A metric measures what it was designed for; a human objection outside that dimension is a gap, not an error.
People think: More annotators solve label noise.
Actually: They reduce variance only with a clear rubric and adjudication; without those, they add cost and confusion.
Interview angle
Strong answers find concrete disagreements, classify the cause and change one variable. They check agreement before blaming the judge. Weak answers argue about the score or immediately propose a new metric.
Sources
- Zheng et al., Judging LLM-as-a-Judge — judge bias and agreement with human raters.
- Es et al., RAGAS — metric assumptions in reference-free evaluation.
- Liang et al., HELM — why one aggregate number hides scenario differences.