Why it matters
Evaluation work is easy to make look rigorous and hard to make trustworthy. A take-home that reports a single score invites the question of what was labelled, how, and whether the metric would move for a real improvement. Reviewers look for sampling rationale, label definitions, agreement checks and a failure analysis. The harness matters less than whether the numbers would change a decision.
Mental model
An eval is a measuring instrument: the dataset is the calibration, the metric is the scale. The metaphor breaks because instruments are calibrated against a known quantity, while labels are human judgements that disagree with each other, so agreement has to be measured rather than assumed.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A calibration plot: for each judge-score bin, the share of items humans pass. The dashed diagonal is perfect calibration. Two slices are shown because a suite-level agreement number can clear a bar while a hard slice sits far below it. Counts are illustrative.
What to notice. Across the whole suite the judge's bins sit 5.0 points from the human pass rate on average, which looks acceptable. On the hard slice the same judge is 15.0 points below the diagonal: at a 0.90 score it reports 90% where humans pass 79%. A single agreement average would have hidden the slice where the judge is wrong.
Text version
A calibration plot from 0 to 1 on the judge score and 0 to 100% on the share of items humans pass, with a dashed perfect-calibration diagonal. In the all-items view nine points track the diagonal within 5 points on average: judge 0.10 maps to 5% human pass, 0.50 to 48%, and 0.90 to 85%. A second button switches to the hard slice, where the same nine scores sit below the diagonal: 0.10 maps to 2%, 0.50 to 31%, and 0.90 to 79%, a mean gap of 15 points. A panel reports the mean gap, the slice's share of items and the gap at the 0.90 score; the marker at the last point prints the human pass rate, and the stat panel names the gap at 0.90.
- State the decision the eval supports, such as choosing between two prompts or gating a release.
- Build a labelled set with an explicit sampling rule, covering head cases, tail cases and cases with no correct answer.
- Define the label and the rubric in writing, then have a second person label a sample to measure agreement.
- Pick a metric that reflects the decision, and report per-slice results alongside the aggregate.
- If a model judges, compare it with human labels on the same sample and report the disagreement cases.
- Write a failure analysis with two examples the system gets wrong and what they suggest.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Human labels | Trustworthy and nuanced | Slow and expensive | The set must be refreshed often |
| Model judge | Cheap and fast at scale | Bias and unverified agreement | The judgement is subtle or domain-specific |
| Exact match | Objective and reproducible | Misses valid paraphrases | Output is free-form |
| Pairwise comparison | Stable for preference | Needs many comparisons and a baseline | You need an absolute quality threshold |
Good for
Trustworthy and nuanced
Cost
Slow and expensive
Fails when
The set must be refreshed often
- The set must be refreshed often
- The judgement is subtle or domain-specific
- Output is free-form
- You need an absolute quality threshold
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Human labels is good for Trustworthy and nuanced, costs Slow and expensive, and fails when The set must be refreshed often. Model judge is good for Cheap and fast at scale, costs Bias and unverified agreement, and fails when The judgement is subtle or domain-specific. Exact match is good for Objective and reproducible, costs Misses valid paraphrases, and fails when Output is free-form. Pairwise comparison is good for Stable for preference, costs Needs many comparisons and a baseline, and fails when You need an absolute quality threshold
Failure modes
- Unlabelled metric. A score is reported without the sampling rule or label definition. Debug: ask how many examples were labelled, by whom and with what instructions.
- Judge agreement unmeasured. The model judge disagrees with humans on the hard slice and nobody knows. Debug: compute agreement on a human-labelled sample and list the disagreements.
- Aggregate only. The average improves while the important slice regresses. Debug: split results by the slice the product cares about and compare.
In production
Version the labelled set and the rubric alongside the harness, and re-check judge agreement whenever the judge model changes. Keep a small held-out slice that is never used for iteration.
Common misconceptions
People think: A higher score means a better system.
Actually: Only if the metric matches the decision and the set resembles production traffic.
People think: An LLM judge removes the need for human labels.
Actually: It needs human labels to establish that it agrees; without that, it is an unvalidated metric.
Interview angle
Strong answers start from the decision, define labels and sampling, and validate any judge against humans. Weak answers report a number without saying what was labelled or whether it would change a choice.
Sources
- Es et al., RAGAS — reference-free metrics and their assumptions.
- Zheng et al., Judging LLM-as-a-Judge — judge agreement, position bias and its limits.
- Liang et al., HELM — multi-metric evaluation and coverage trade-offs.