Why it matters
A single quality score compresses several independent properties into one number, and those properties move in different directions. An answer can be correct but incomplete, complete but ungrounded, or grounded but useless for the task. Product decisions need to know which dimension moved, because the fix for a grounding problem is retrieval while the fix for a completeness problem is the prompt or the task decomposition.
Mental model
It is a medical report: accurate findings, complete history, a clear recommendation. The metaphor breaks because readers weight those dimensions differently depending on what they came for, and the same report can be excellent for one reader and useless for another.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Modelled 0-100 score distributions: one weighted composite, or the four dimensions behind it, against the same 70-point bar. The composite is a 45/25/20/10 average of the four medians, not a fifth measurement. Means and spreads are illustrative; the masking is the claim.
What to notice. The composite sits at 75.4 and clears a 70-point bar for 78% of answers. Switch to per dimension and the same answers show completeness at a median of 58 and usefulness at 66, both below the bar, while correctness is at 86. Tracking the composite alone would call the release healthy; the per-dimension view names the fix.
Text version
Score distributions from 10 to 100 against a 70-point bar. In the composite view one narrow curve peaks at 75.4 and 22% of answers fall below the bar. In the per-dimension view four curves peak at different places: correctness at 86 with 0.4% below the bar, grounding at 74 with 34.5%, completeness at 58 with 84.1%, and usefulness at 66 with 61.2%. A two-button control switches between the composite and the four dimensions; a panel reports that two of four dimension medians sit below the bar while the composite is above it.
- Define the dimensions for the product: factual correctness, coverage of the ask, grounding, format and actionability.
- Write one testable question per dimension and a rubric level for failure.
- Score factual claims against evidence separately from style and tone.
- Aggregate only after reporting each dimension, weighting by the product's tolerance for each failure.
- Sample cases where judge and human disagree and inspect the dimension that split.
- Track each dimension over releases rather than the composite.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Composite score | Easy to track | Hides which dimension moved | You need to diagnose a regression |
| Per-dimension scores | Actionable | More annotation effort | Leadership wants one number |
| Claim-level factuality | Precise | Pipeline complexity | Answer is advisory rather than factual |
| Human usefulness rating | Captures intent | Expensive and subjective | Volume is high |
Good for
Easy to track
Cost
Hides which dimension moved
Fails when
You need to diagnose a regression
- You need to diagnose a regression
- Leadership wants one number
- Answer is advisory rather than factual
- Volume is high
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Composite score is good for Easy to track, costs Hides which dimension moved, and fails when You need to diagnose a regression. Per-dimension scores is good for Actionable, costs More annotation effort, and fails when Leadership wants one number. Claim-level factuality is good for Precise, costs Pipeline complexity, and fails when Answer is advisory rather than factual. Human usefulness rating is good for Captures intent, costs Expensive and subjective, and fails when Volume is high
Failure modes
- Fluent wrong answer. A confident, well-structured response passes a style-aware judge. Debug: extract claims and check each against the retrieved evidence.
- Completeness blind spot. The answer is correct but ignores half the request. Debug: score coverage of each sub-question in the prompt separately.
- Composite masking. The overall score is flat while grounding falls and style rises. Debug: plot dimension scores per release.
In production
Track dimensions separately with sampled human labels. Alert when any dimension crosses its threshold, not only when the composite moves.
Common misconceptions
People think: A single quality score is enough for release decisions.
Actually: It hides offsetting changes between dimensions.
People think: Correct answers are also useful answers.
Actually: Usefulness depends on completeness, format and audience, which correctness does not cover.
Interview angle
Decompose quality, name a measurement per dimension and explain how the composite misleads. A strong answer connects each dimension to a different fix. Defining quality as accuracy, with no completeness or grounding, is the weak version.
Sources
- Liu et al., G-Eval — criteria-based scoring of generated answers.
- Min et al., FActScore — atomic claim checking for factual precision.
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench — multi-turn answer quality and judge agreement.