Why it matters
Quality is multi-dimensional, and asking whether an answer is good collapses correctness, completeness and tone into one impression. A rubric separates those dimensions and defines what each level looks like, which makes scores comparable across annotators, across releases and between a human and a model judge. Without anchored levels, every rater invents a private scale and the numbers look precise while measuring nothing shared.
Mental model
It is a marking scheme for an exam. The metaphor breaks because exam answers are closed, while a good answer may satisfy the criteria in an order the rubric never anticipated, so raters must judge coverage rather than match a model answer.
How it works
- 1List the dimensions that matter separately: correctness, completeness, grounding, format and safety.
- 2Write each level as an observable property with an example output at the boundary.
- 3Test the rubric on twenty outputs with two raters and revise lines that produce disagreement.
- 4Order the criteria so hard failures are checked before quality dimensions.
- 5Version the rubric and attach the version to every stored score.
- 6Re-validate when the product, output format or task mix changes.
Step 1 of 6
List the dimensions that matter separately: correctness, completeness, grounding, format and safety.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. List the dimensions that matter separately: correctness, completeness, grounding, format … 2. Write each level as an observable property with an example output at the boundary. 3. Test the rubric on twenty outputs with two raters and revise lines that produce disagreem… 4. Order the criteria so hard failures are checked before quality dimensions. 5. Version the rubric and attach the version to every stored score. 6. Re-validate when the product, output format or task mix changes.
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. The share of items each rater places at each level of a five-level rubric, for two human raters and a model judge, before and after every level gets an example output. The same scale is used differently until the anchors are written; the distribution, not the average, shows where.
What to notice. Without anchors the human raters put 54% of items at levels 4 and 5, while the judge puts 74% there and only 2% at level 1 against the humans' 8%. The two distributions differ by 20 points of mass, which an average score near the same value would hide. With an example at each level the judge's top-level share falls to 19% and the distance to 3 points, so every level is within three points of the human share.
Text version
Two horizontal bar series across five rubric levels, one for the consensus of two human raters and one for a model judge. The human shares are 8%, 12%, 26%, 35% and 19% from level 1 to level 5. The unanchored judge is 2%, 6%, 18%, 36% and 38%: it overuses the top level and almost never uses the bottom. A button switches to an anchored judge whose shares are 7%, 11%, 25%, 38% and 19%, close to the human distribution. A panel reports the level-5 share for each rater, the distance between the distributions and how many levels sit within three points.
- List the dimensions that matter separately: correctness, completeness, grounding, format and safety.
- Write each level as an observable property with an example output at the boundary.
- Test the rubric on twenty outputs with two raters and revise lines that produce disagreement.
- Order the criteria so hard failures are checked before quality dimensions.
- Version the rubric and attach the version to every stored score.
- Re-validate when the product, output format or task mix changes.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Few coarse dimensions | Fast and reliable | Loses nuance | Quality differences are subtle |
| Many fine dimensions | Rich signal | Rater fatigue, correlated scores | Scale is large |
| Binary pass criteria | Unambiguous | No gradation | Partial quality matters |
| Holistic single score | Cheap | Hides which dimension failed | You need to act on the result |
Good for
Fast and reliable
Cost
Loses nuance
Fails when
Quality differences are subtle
- Quality differences are subtle
- Scale is large
- Partial quality matters
- You need to act on the result
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Few coarse dimensions is good for Fast and reliable, costs Loses nuance, and fails when Quality differences are subtle. Many fine dimensions is good for Rich signal, costs Rater fatigue, correlated scores, and fails when Scale is large. Binary pass criteria is good for Unambiguous, costs No gradation, and fails when Partial quality matters. Holistic single score is good for Cheap, costs Hides which dimension failed, and fails when You need to act on the result
Failure modes
- Double counting. One factual error lowers correctness, grounding and clarity at once. Debug: correlate dimension scores and inspect near-perfect pairs.
- Unanchored levels. Annotators read adjectives differently and scores drift. Debug: compare level distributions between annotators on the same items.
- Rubric drift. The rubric changed mid-dataset and scores are no longer comparable. Debug: check the rubric version stored with each score.
In production
Track score distribution per dimension and agreement per criterion. Re-run the calibration set after every rubric edit and retire criteria that never vary.
Common misconceptions
People think: A detailed rubric is automatically more reliable.
Actually: Detail without anchors adds reading time and interpretation variance.
People think: Rubrics make evaluation objective.
Actually: They make it explicit and repeatable, while the judgement is still made by a person or a model.
Interview angle
Define independent dimensions, anchor levels with examples and test whether raters apply them consistently. A strong answer covers hard-failure ordering and versioning. A list of adjectives such as clear and helpful, never tested across two raters, is the weak answer.
Sources
- Liu et al., G-Eval — criteria-driven scoring with explicit evaluation steps.
- Ribeiro et al., Beyond Accuracy with CheckList — turning desired behaviours into testable criteria.
- Kim et al., Prometheus — an open evaluator model driven by rubrics.