Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 32 of 33

Troubleshoot: eval vs human

0 of 33 complete

Why it matters

An eval is a proxy, and a proxy that disagrees with the people using the product is worse than no eval because it creates false confidence. The disagreement has a small number of causes: the labelled set does not resemble production, the rubric is ambiguous, the judge has a systematic bias, or the humans are reacting to something the metric does not measure, such as tone or latency. Each cause needs a different investigation, so the first step is to find examples rather than debate the score.

Mental model

It is a smoke alarm that stays silent during a fire: either the sensor is pointed elsewhere, the threshold is wrong, or it is measuring a different kind of smoke. The metaphor breaks because humans also disagree with each other, so there is no single ground truth to calibrate against.

How it works

Mechanism, step by stepInteractive
  1. 1Collect ten cases where the eval scored well and a human rated the output poorly.
  2. 2Read the outputs and write what the human objected to, in their words.
  3. 3Check whether the objection is covered by the rubric; if not, the metric is measuring something else.
  4. 4Measure inter-rater agreement on a sample, since low agreement means the label is unstable.
  5. 5Compare the eval set's distribution with production traffic on length, topic and query type.
  6. 6Change one thing, either the rubric, the judge prompt or the sample, and re-measure agreement on the same cases.

Step 1 of 6

Collect ten cases where the eval scored well and a human rated the output poorly.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Collect ten cases where the eval scored well and a human rated the output poorly. 2. Read the outputs and write what the human objected to, in their words. 3. Check whether the objection is covered by the rubric; if not, the metric is measuring som… 4. Measure inter-rater agreement on a sample, since low agreement means the label is unstabl… 5. Compare the eval set's distribution with production traffic on length, topic and query ty… 6. Change one thing, either the rubric, the judge prompt or the sample, and re-measure agree…

  1. Collect ten cases where the eval scored well and a human rated the output poorly.
  2. Read the outputs and write what the human objected to, in their words.
  3. Check whether the objection is covered by the rubric; if not, the metric is measuring something else.
  4. Measure inter-rater agreement on a sample, since low agreement means the label is unstable.
  5. Compare the eval set's distribution with production traffic on length, topic and query type.
  6. Change one thing, either the rubric, the judge prompt or the sample, and re-measure agreement on the same cases.

Trade-offs

ChoiceGood forCostFails when
Trust the humansCatches what the metric missesHuman judgement is noisy and expensiveThe complaint comes from one vocal user
Trust the evalConsistent and cheapSystematic blind spots persistThe rubric was written before the product changed
Add a new metricCovers the missing dimensionMore numbers to reconcileNobody agrees which metric decides
Re-weight the setReflects productionOld regressions become invisibleThe tail is the part that matters
Trade-off mapInteractive

Good for

Catches what the metric misses

Cost

Human judgement is noisy and expensive

Fails when

The complaint comes from one vocal user

  1. The complaint comes from one vocal user
  2. The rubric was written before the product changed
  3. Nobody agrees which metric decides
  4. The tail is the part that matters

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Trust the humans is good for Catches what the metric misses, costs Human judgement is noisy and expensive, and fails when The complaint comes from one vocal user. Trust the eval is good for Consistent and cheap, costs Systematic blind spots persist, and fails when The rubric was written before the product changed. Add a new metric is good for Covers the missing dimension, costs More numbers to reconcile, and fails when Nobody agrees which metric decides. Re-weight the set is good for Reflects production, costs Old regressions become invisible, and fails when The tail is the part that matters

Failure modes

  • Rubric gap. Humans penalise tone and the metric only checks factuality. Debug: read ten disagreements and tag the objection against the rubric dimensions.
  • Label noise. Two annotators disagree and the eval is chasing an unstable target. Debug: compute agreement between annotators on the same sample.
  • Distribution shift. The eval set is drawn from launch traffic and production has moved on. Debug: compare query length and topic distributions between the set and last week's logs.

In production

Track eval scores and human review outcomes on the same dashboard, and alert when they diverge for a week. Keep a small freshly sampled set to detect drift.

Common misconceptions

People think: If the eval is well built, human disagreement means the humans are wrong.

Actually: A metric measures what it was designed for; a human objection outside that dimension is a gap, not an error.

People think: More annotators solve label noise.

Actually: They reduce variance only with a clear rubric and adjudication; without those, they add cost and confusion.

Interview angle

Strong answers find concrete disagreements, classify the cause and change one variable. They check agreement before blaming the judge. Weak answers argue about the score or immediately propose a new metric.

Sources