Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 14 of 28

Answer quality

0 of 28 complete

Why it matters

A single quality score compresses several independent properties into one number, and those properties move in different directions. An answer can be correct but incomplete, complete but ungrounded, or grounded but useless for the task. Product decisions need to know which dimension moved, because the fix for a grounding problem is retrieval while the fix for a completeness problem is the prompt or the task decomposition.

Mental model

It is a medical report: accurate findings, complete history, a clear recommendation. The metaphor breaks because readers weight those dimensions differently depending on what they came for, and the same report can be excellent for one reader and useless for another.

How it works

The composite hides twoInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Modelled 0-100 score distributions: one weighted composite, or the four dimensions behind it, against the same 70-point bar. The composite is a 45/25/20/10 average of the four medians, not a fifth measurement. Means and spreads are illustrative; the masking is the claim.

What to notice. The composite sits at 75.4 and clears a 70-point bar for 78% of answers. Switch to per dimension and the same answers show completeness at a median of 58 and usefulness at 66, both below the bar, while correctness is at 86. Tracking the composite alone would call the release healthy; the per-dimension view names the fix.

Text version

Score distributions from 10 to 100 against a 70-point bar. In the composite view one narrow curve peaks at 75.4 and 22% of answers fall below the bar. In the per-dimension view four curves peak at different places: correctness at 86 with 0.4% below the bar, grounding at 74 with 34.5%, completeness at 58 with 84.1%, and usefulness at 66 with 61.2%. A two-button control switches between the composite and the four dimensions; a panel reports that two of four dimension medians sit below the bar while the composite is above it.

  1. Define the dimensions for the product: factual correctness, coverage of the ask, grounding, format and actionability.
  2. Write one testable question per dimension and a rubric level for failure.
  3. Score factual claims against evidence separately from style and tone.
  4. Aggregate only after reporting each dimension, weighting by the product's tolerance for each failure.
  5. Sample cases where judge and human disagree and inspect the dimension that split.
  6. Track each dimension over releases rather than the composite.

Trade-offs

ChoiceGood forCostFails when
Composite scoreEasy to trackHides which dimension movedYou need to diagnose a regression
Per-dimension scoresActionableMore annotation effortLeadership wants one number
Claim-level factualityPrecisePipeline complexityAnswer is advisory rather than factual
Human usefulness ratingCaptures intentExpensive and subjectiveVolume is high
Trade-off mapInteractive

Good for

Easy to track

Cost

Hides which dimension moved

Fails when

You need to diagnose a regression

  1. You need to diagnose a regression
  2. Leadership wants one number
  3. Answer is advisory rather than factual
  4. Volume is high

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Composite score is good for Easy to track, costs Hides which dimension moved, and fails when You need to diagnose a regression. Per-dimension scores is good for Actionable, costs More annotation effort, and fails when Leadership wants one number. Claim-level factuality is good for Precise, costs Pipeline complexity, and fails when Answer is advisory rather than factual. Human usefulness rating is good for Captures intent, costs Expensive and subjective, and fails when Volume is high

Failure modes

  • Fluent wrong answer. A confident, well-structured response passes a style-aware judge. Debug: extract claims and check each against the retrieved evidence.
  • Completeness blind spot. The answer is correct but ignores half the request. Debug: score coverage of each sub-question in the prompt separately.
  • Composite masking. The overall score is flat while grounding falls and style rises. Debug: plot dimension scores per release.

In production

Track dimensions separately with sampled human labels. Alert when any dimension crosses its threshold, not only when the composite moves.

Common misconceptions

People think: A single quality score is enough for release decisions.

Actually: It hides offsetting changes between dimensions.

People think: Correct answers are also useful answers.

Actually: Usefulness depends on completeness, format and audience, which correctness does not cover.

Interview angle

Decompose quality, name a measurement per dimension and explain how the composite misleads. A strong answer connects each dimension to a different fix. Defining quality as accuracy, with no completeness or grounding, is the weak version.

Sources