Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 11 of 28

Judge calibration

0 of 28 complete

Why it matters

A judge returns a score, not a truth value, and the mapping from score to decision has to be learned from human labels. That mapping is not uniform: a judge may be reliable on short factual answers and unreliable on multi-step reasoning, so a single threshold hides the slice where it fails. Calibration is the work of measuring agreement, choosing thresholds for the decision at hand and knowing when to escalate to a person.

Mental model

It is tuning a scale against known weights. The metaphor breaks because the judge's own confidence is not a reading you can trust, and the weights drift when the task mix changes, so calibration is a repeated procedure rather than a one-time setting.

How it works

Mechanism, step by stepInteractive
  1. 1Build a human-labelled calibration set of a few hundred items spanning the categories you ship.
  2. 2Collect judge scores on the same items with the exact prompt and model you will deploy.
  3. 3Compute agreement overall and per category, and plot judge scores against human labels.
  4. 4Choose thresholds from the precision-recall trade-off for the decision, not from the midpoint of the scale.
  5. 5Define an abstain or escalate band where judge and human disagree most.
  6. 6Re-run the calibration set on a schedule and after every judge change.

Step 1 of 6

Build a human-labelled calibration set of a few hundred items spanning the categories you ship.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Build a human-labelled calibration set of a few hundred items spanning the categories you… 2. Collect judge scores on the same items with the exact prompt and model you will deploy. 3. Compute agreement overall and per category, and plot judge scores against human labels. 4. Choose thresholds from the precision-recall trade-off for the decision, not from the midp… 5. Define an abstain or escalate band where judge and human disagree most. 6. Re-run the calibration set on a schedule and after every judge change.

  1. Build a human-labelled calibration set of a few hundred items spanning the categories you ship.
  2. Collect judge scores on the same items with the exact prompt and model you will deploy.
  3. Compute agreement overall and per category, and plot judge scores against human labels.
  4. Choose thresholds from the precision-recall trade-off for the decision, not from the midpoint of the scale.
  5. Define an abstain or escalate band where judge and human disagree most.
  6. Re-run the calibration set on a schedule and after every judge change.

Trade-offs

ChoiceGood forCostFails when
Global thresholdSimple to operateMiscalibrated for some categoriesError rates vary by slice
Per-category thresholdsBetter decisionsMore configuration, thinner samplesCategories are small
Judge with escalation bandRoutes uncertain cases to humansHuman loadVolume is high
Human-only labellingTrustworthyCost and latencyScale matters
Trade-off mapInteractive

Good for

Simple to operate

Cost

Miscalibrated for some categories

Fails when

Error rates vary by slice

  1. Error rates vary by slice
  2. Categories are small
  3. Volume is high
  4. Scale matters

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Global threshold is good for Simple to operate, costs Miscalibrated for some categories, and fails when Error rates vary by slice. Per-category thresholds is good for Better decisions, costs More configuration, thinner samples, and fails when Categories are small. Judge with escalation band is good for Routes uncertain cases to humans, costs Human load, and fails when Volume is high. Human-only labelling is good for Trustworthy, costs Cost and latency, and fails when Scale matters

Failure modes

  • Calibration on the wrong set. The judge looks aligned on curated examples and fails on production phrasing. Debug: compare the calibration set's score distribution with a production sample.
  • Threshold overfit. The cutoff is tuned to the calibration items and does not transfer. Debug: hold out a portion and recompute agreement.
  • Silent judge upgrade. The provider updates the model and agreement drops without a prompt change. Debug: pin the judge version and re-run calibration on upgrade.

In production

Store the calibration curve with the judge version and alert when live agreement on sampled items falls. Keep the human-labelled set frozen and versioned.

Common misconceptions

People think: A judge that agrees with humans is calibrated everywhere.

Actually: Agreement is slice-specific, and a strong average can hide a category where it fails.

People think: Judge confidence can stand in for calibration.

Actually: Model-reported confidence is not a probability and must be validated against outcomes.

Interview angle

Describe a labelled calibration set, per-category agreement and thresholds chosen from a decision trade-off. A strong answer covers escalation and version pinning. Assuming a judge score is a probability, with no agreement measurement, is the weak version.

Sources