Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 7 of 28

Human evaluation

0 of 28 complete

Why it matters

Automated metrics can compare wording or structure but cannot decide whether an answer helped someone finish a task. Human judgement supplies that ground truth, and it brings its own variance: different people apply different standards, tire at different rates and infer intent differently. The engineering problem is to make the judgement repeatable enough that a difference between two systems is attributable to the systems.

Mental model

It is a tasting panel: trained, blinded and calibrated. The metaphor breaks because tasters agree on saltiness and disagree about whether the dish is good, and the second disagreement is the one the product needs settled before anyone ships a release.

How it works

The middle of the scale is the noisy partInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Exact agreement between two raters at each rubric level, before and after the scale anchors were rewritten. Agreement is conditional on the items placed at that level, so the ends look better than the middle. Counts are illustrative.

What to notice. Before the anchors, the two raters agree exactly on 58% of items overall and agreement bottoms at 41% on level 3. After rewriting the anchors it is 77% overall and level 3 reaches 69%. The ends score higher on this sample, where each holds 10% of items; the middle is where the rubric asks for a judgement rather than a verdict.

Text version

Exact agreement between two raters from level 1 to level 5 on a five-point rubric, against an agreement bar at 70%. Before the rewrite the stepped line reads 88%, 66%, 41%, 55% and 82%, holds the last value to the plot edge, and marks the trough at level 3; three levels sit below the bar. A second button adds the after-rewrite line: 92%, 80%, 69%, 74% and 88%, with one level below the bar. A panel reports overall exact agreement, the worst level and how many levels fall below 70%.

  1. Define the judgement in one sentence and write scale levels with anchored examples.
  2. Pilot on a small batch, then rewrite every rubric line where annotators diverge.
  3. Randomise presentation order and hide which system produced which output.
  4. Collect at least two judgements per item and compute agreement before averaging.
  5. Record time per item so the cost of the next round is estimable.
  6. Reserve adjudication for disagreements that cross a decision boundary.

Trade-offs

ChoiceGood forCostFails when
Expert annotatorsNuanced and domain-correctExpensive and scarceVolume is large
Crowd annotatorsCheap and fastShallow judgement, needs checksTask needs domain knowledge
Pairwise preferenceEasier than absolute scoresNo absolute quality, order effectsYou need a calibrated score
Absolute rubric scoreComparable across systemsAnchors are hard to keep stableAnnotators interpret scales differently
Trade-off mapInteractive

Good for

Nuanced and domain-correct

Cost

Expensive and scarce

Fails when

Volume is large

  1. Volume is large
  2. Task needs domain knowledge
  3. You need a calibrated score
  4. Annotators interpret scales differently

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Expert annotators is good for Nuanced and domain-correct, costs Expensive and scarce, and fails when Volume is large. Crowd annotators is good for Cheap and fast, costs Shallow judgement, needs checks, and fails when Task needs domain knowledge. Pairwise preference is good for Easier than absolute scores, costs No absolute quality, order effects, and fails when You need a calibrated score. Absolute rubric score is good for Comparable across systems, costs Anchors are hard to keep stable, and fails when Annotators interpret scales differently

Failure modes

  • Rubric ambiguity. Two annotators apply different standards and the mean is meaningless. Debug: read the disagreement cases and check whether they cite different rubric lines.
  • Order effect. The first output is preferred regardless of quality. Debug: compare win rates when the positions are swapped.
  • Annotator drift. Standards shift across a long session. Debug: re-score a fixed calibration set at the start and end of each batch.

In production

Track agreement, time per item and calibration-set scores per batch. Cap batch length and re-calibrate whenever a new annotator joins.

Common misconceptions

People think: Human judgement is objective ground truth.

Actually: It is the reference standard, and it carries bias, fatigue and interpretation differences.

People think: More annotators always improves the estimate.

Actually: Beyond a point, added annotators reduce variance slowly while cost grows linearly, and rubric quality dominates.

Interview angle

Treat human judgement as an instrument: rubric first, blinding, repeated judgements and reported agreement. A strong answer mentions cost per item and adjudication rules. Asking colleagues to rate outputs, with no anchors or order control, is the weak version.

Sources