Why it matters
Automated metrics can compare wording or structure but cannot decide whether an answer helped someone finish a task. Human judgement supplies that ground truth, and it brings its own variance: different people apply different standards, tire at different rates and infer intent differently. The engineering problem is to make the judgement repeatable enough that a difference between two systems is attributable to the systems.
Mental model
It is a tasting panel: trained, blinded and calibrated. The metaphor breaks because tasters agree on saltiness and disagree about whether the dish is good, and the second disagreement is the one the product needs settled before anyone ships a release.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Exact agreement between two raters at each rubric level, before and after the scale anchors were rewritten. Agreement is conditional on the items placed at that level, so the ends look better than the middle. Counts are illustrative.
What to notice. Before the anchors, the two raters agree exactly on 58% of items overall and agreement bottoms at 41% on level 3. After rewriting the anchors it is 77% overall and level 3 reaches 69%. The ends score higher on this sample, where each holds 10% of items; the middle is where the rubric asks for a judgement rather than a verdict.
Text version
Exact agreement between two raters from level 1 to level 5 on a five-point rubric, against an agreement bar at 70%. Before the rewrite the stepped line reads 88%, 66%, 41%, 55% and 82%, holds the last value to the plot edge, and marks the trough at level 3; three levels sit below the bar. A second button adds the after-rewrite line: 92%, 80%, 69%, 74% and 88%, with one level below the bar. A panel reports overall exact agreement, the worst level and how many levels fall below 70%.
- Define the judgement in one sentence and write scale levels with anchored examples.
- Pilot on a small batch, then rewrite every rubric line where annotators diverge.
- Randomise presentation order and hide which system produced which output.
- Collect at least two judgements per item and compute agreement before averaging.
- Record time per item so the cost of the next round is estimable.
- Reserve adjudication for disagreements that cross a decision boundary.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Expert annotators | Nuanced and domain-correct | Expensive and scarce | Volume is large |
| Crowd annotators | Cheap and fast | Shallow judgement, needs checks | Task needs domain knowledge |
| Pairwise preference | Easier than absolute scores | No absolute quality, order effects | You need a calibrated score |
| Absolute rubric score | Comparable across systems | Anchors are hard to keep stable | Annotators interpret scales differently |
Good for
Nuanced and domain-correct
Cost
Expensive and scarce
Fails when
Volume is large
- Volume is large
- Task needs domain knowledge
- You need a calibrated score
- Annotators interpret scales differently
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Expert annotators is good for Nuanced and domain-correct, costs Expensive and scarce, and fails when Volume is large. Crowd annotators is good for Cheap and fast, costs Shallow judgement, needs checks, and fails when Task needs domain knowledge. Pairwise preference is good for Easier than absolute scores, costs No absolute quality, order effects, and fails when You need a calibrated score. Absolute rubric score is good for Comparable across systems, costs Anchors are hard to keep stable, and fails when Annotators interpret scales differently
Failure modes
- Rubric ambiguity. Two annotators apply different standards and the mean is meaningless. Debug: read the disagreement cases and check whether they cite different rubric lines.
- Order effect. The first output is preferred regardless of quality. Debug: compare win rates when the positions are swapped.
- Annotator drift. Standards shift across a long session. Debug: re-score a fixed calibration set at the start and end of each batch.
In production
Track agreement, time per item and calibration-set scores per batch. Cap batch length and re-calibrate whenever a new annotator joins.
Common misconceptions
People think: Human judgement is objective ground truth.
Actually: It is the reference standard, and it carries bias, fatigue and interpretation differences.
People think: More annotators always improves the estimate.
Actually: Beyond a point, added annotators reduce variance slowly while cost grows linearly, and rubric quality dominates.
Interview angle
Treat human judgement as an instrument: rubric first, blinding, repeated judgements and reported agreement. A strong answer mentions cost per item and adjudication rules. Asking colleagues to rate outputs, with no anchors or order control, is the weak version.
Sources
- Howcroft et al., Evaluation of Text Generation: A Survey — human evaluation designs and their pitfalls.
- Chiang et al., Chatbot Arena — large-scale pairwise human preference collection.
- Artstein and Poesio, Inter-Coder Agreement for Computational Linguistics — why agreement must accompany human labels.