Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 22 of 33

Take-home: evals

0 of 33 complete

Why it matters

Evaluation work is easy to make look rigorous and hard to make trustworthy. A take-home that reports a single score invites the question of what was labelled, how, and whether the metric would move for a real improvement. Reviewers look for sampling rationale, label definitions, agreement checks and a failure analysis. The harness matters less than whether the numbers would change a decision.

Mental model

An eval is a measuring instrument: the dataset is the calibration, the metric is the scale. The metaphor breaks because instruments are calibrated against a known quantity, while labels are human judgements that disagree with each other, so agreement has to be measured rather than assumed.

How it works

Agreement is not uniformInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. A calibration plot: for each judge-score bin, the share of items humans pass. The dashed diagonal is perfect calibration. Two slices are shown because a suite-level agreement number can clear a bar while a hard slice sits far below it. Counts are illustrative.

What to notice. Across the whole suite the judge's bins sit 5.0 points from the human pass rate on average, which looks acceptable. On the hard slice the same judge is 15.0 points below the diagonal: at a 0.90 score it reports 90% where humans pass 79%. A single agreement average would have hidden the slice where the judge is wrong.

Text version

A calibration plot from 0 to 1 on the judge score and 0 to 100% on the share of items humans pass, with a dashed perfect-calibration diagonal. In the all-items view nine points track the diagonal within 5 points on average: judge 0.10 maps to 5% human pass, 0.50 to 48%, and 0.90 to 85%. A second button switches to the hard slice, where the same nine scores sit below the diagonal: 0.10 maps to 2%, 0.50 to 31%, and 0.90 to 79%, a mean gap of 15 points. A panel reports the mean gap, the slice's share of items and the gap at the 0.90 score; the marker at the last point prints the human pass rate, and the stat panel names the gap at 0.90.

  1. State the decision the eval supports, such as choosing between two prompts or gating a release.
  2. Build a labelled set with an explicit sampling rule, covering head cases, tail cases and cases with no correct answer.
  3. Define the label and the rubric in writing, then have a second person label a sample to measure agreement.
  4. Pick a metric that reflects the decision, and report per-slice results alongside the aggregate.
  5. If a model judges, compare it with human labels on the same sample and report the disagreement cases.
  6. Write a failure analysis with two examples the system gets wrong and what they suggest.

Trade-offs

ChoiceGood forCostFails when
Human labelsTrustworthy and nuancedSlow and expensiveThe set must be refreshed often
Model judgeCheap and fast at scaleBias and unverified agreementThe judgement is subtle or domain-specific
Exact matchObjective and reproducibleMisses valid paraphrasesOutput is free-form
Pairwise comparisonStable for preferenceNeeds many comparisons and a baselineYou need an absolute quality threshold
Trade-off mapInteractive

Good for

Trustworthy and nuanced

Cost

Slow and expensive

Fails when

The set must be refreshed often

  1. The set must be refreshed often
  2. The judgement is subtle or domain-specific
  3. Output is free-form
  4. You need an absolute quality threshold

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Human labels is good for Trustworthy and nuanced, costs Slow and expensive, and fails when The set must be refreshed often. Model judge is good for Cheap and fast at scale, costs Bias and unverified agreement, and fails when The judgement is subtle or domain-specific. Exact match is good for Objective and reproducible, costs Misses valid paraphrases, and fails when Output is free-form. Pairwise comparison is good for Stable for preference, costs Needs many comparisons and a baseline, and fails when You need an absolute quality threshold

Failure modes

  • Unlabelled metric. A score is reported without the sampling rule or label definition. Debug: ask how many examples were labelled, by whom and with what instructions.
  • Judge agreement unmeasured. The model judge disagrees with humans on the hard slice and nobody knows. Debug: compute agreement on a human-labelled sample and list the disagreements.
  • Aggregate only. The average improves while the important slice regresses. Debug: split results by the slice the product cares about and compare.

In production

Version the labelled set and the rubric alongside the harness, and re-check judge agreement whenever the judge model changes. Keep a small held-out slice that is never used for iteration.

Common misconceptions

People think: A higher score means a better system.

Actually: Only if the metric matches the decision and the set resembles production traffic.

People think: An LLM judge removes the need for human labels.

Actually: It needs human labels to establish that it agrees; without that, it is an unvalidated metric.

Interview angle

Strong answers start from the decision, define labels and sampling, and validate any judge against humans. Weak answers report a number without saying what was labelled or whether it would change a choice.

Sources