Why it matters
A judge returns a score, not a truth value, and the mapping from score to decision has to be learned from human labels. That mapping is not uniform: a judge may be reliable on short factual answers and unreliable on multi-step reasoning, so a single threshold hides the slice where it fails. Calibration is the work of measuring agreement, choosing thresholds for the decision at hand and knowing when to escalate to a person.
Mental model
It is tuning a scale against known weights. The metaphor breaks because the judge's own confidence is not a reading you can trust, and the weights drift when the task mix changes, so calibration is a repeated procedure rather than a one-time setting.
How it works
- 1Build a human-labelled calibration set of a few hundred items spanning the categories you ship.
- 2Collect judge scores on the same items with the exact prompt and model you will deploy.
- 3Compute agreement overall and per category, and plot judge scores against human labels.
- 4Choose thresholds from the precision-recall trade-off for the decision, not from the midpoint of the scale.
- 5Define an abstain or escalate band where judge and human disagree most.
- 6Re-run the calibration set on a schedule and after every judge change.
Step 1 of 6
Build a human-labelled calibration set of a few hundred items spanning the categories you ship.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Build a human-labelled calibration set of a few hundred items spanning the categories you… 2. Collect judge scores on the same items with the exact prompt and model you will deploy. 3. Compute agreement overall and per category, and plot judge scores against human labels. 4. Choose thresholds from the precision-recall trade-off for the decision, not from the midp… 5. Define an abstain or escalate band where judge and human disagree most. 6. Re-run the calibration set on a schedule and after every judge change.
- Build a human-labelled calibration set of a few hundred items spanning the categories you ship.
- Collect judge scores on the same items with the exact prompt and model you will deploy.
- Compute agreement overall and per category, and plot judge scores against human labels.
- Choose thresholds from the precision-recall trade-off for the decision, not from the midpoint of the scale.
- Define an abstain or escalate band where judge and human disagree most.
- Re-run the calibration set on a schedule and after every judge change.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Global threshold | Simple to operate | Miscalibrated for some categories | Error rates vary by slice |
| Per-category thresholds | Better decisions | More configuration, thinner samples | Categories are small |
| Judge with escalation band | Routes uncertain cases to humans | Human load | Volume is high |
| Human-only labelling | Trustworthy | Cost and latency | Scale matters |
Good for
Simple to operate
Cost
Miscalibrated for some categories
Fails when
Error rates vary by slice
- Error rates vary by slice
- Categories are small
- Volume is high
- Scale matters
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Global threshold is good for Simple to operate, costs Miscalibrated for some categories, and fails when Error rates vary by slice. Per-category thresholds is good for Better decisions, costs More configuration, thinner samples, and fails when Categories are small. Judge with escalation band is good for Routes uncertain cases to humans, costs Human load, and fails when Volume is high. Human-only labelling is good for Trustworthy, costs Cost and latency, and fails when Scale matters
Failure modes
- Calibration on the wrong set. The judge looks aligned on curated examples and fails on production phrasing. Debug: compare the calibration set's score distribution with a production sample.
- Threshold overfit. The cutoff is tuned to the calibration items and does not transfer. Debug: hold out a portion and recompute agreement.
- Silent judge upgrade. The provider updates the model and agreement drops without a prompt change. Debug: pin the judge version and re-run calibration on upgrade.
In production
Store the calibration curve with the judge version and alert when live agreement on sampled items falls. Keep the human-labelled set frozen and versioned.
Common misconceptions
People think: A judge that agrees with humans is calibrated everywhere.
Actually: Agreement is slice-specific, and a strong average can hide a category where it fails.
People think: Judge confidence can stand in for calibration.
Actually: Model-reported confidence is not a probability and must be validated against outcomes.
Interview angle
Describe a labelled calibration set, per-category agreement and thresholds chosen from a decision trade-off. A strong answer covers escalation and version pinning. Assuming a judge score is a probability, with no agreement measurement, is the weak version.
Sources
- Guo et al., On Calibration of Modern Neural Networks — confidence, calibration error and reliability diagrams.
- Jung et al., Trust or Escalate — judge guarantees and routing uncertain cases to humans.
- Kim et al., Prometheus 2 — an open judge model with rubric-based scoring.