Why it matters
A language model does not return the same output twice, and quality is not one property. An answer can be correct but unreadable, grounded but incomplete, polite but wrong about a number. Change a prompt, a retrieval parameter or a model version and the system moves in several dimensions at once.
Evaluation converts that judgement into a comparison with a denominator. "The new prompt seemed better in five examples" is a demo. "Pass rate on 240 labelled cases moved from 0.81 to 0.86, with an interval that excludes zero, and retrieval metrics were unchanged" is a decision.
The cost asymmetry is the argument. A bad prompt caught offline costs minutes. The same prompt caught by a customer costs trust and an incident review.
Mental model
Think of an aircraft instrument panel, not a single grade. Each gauge measures one quantity, is reliable in a range, and is cross-checked against another before you act. Airspeed alone will not tell you that you are stalling.
The analogy breaks in one important way. A physical gauge is calibrated by its manufacturer. Every scorer you add — a regex, a similarity threshold, a model judge — arrives uncalibrated. Until you compare it with human labels on your own data, precision is not evidence of accuracy.
How it works
- 1Start from the decision. Is the eval there to block a release, choose between prompts, set a routing threshold, or detect drift? The decision fixes the metric, the sample size and the acceptable error. An eval that informs nothing becomes a dashboard nobody reads.
- 2Decompose the system. For RAG, retrieval recall, context precision and answer faithfulness are separate measurements. For an agent, task success, tool-call validity and trajectory length are separate. End-to-end scores gate releases; stage scores explain failures.
- 3Define cases with grading criteria. A case is an input plus the rule that decides whether an output is acceptable: exact match, schema check, rubric or reference answer. Datasets and golden sets covers building one that represents traffic.
- 4Pick a scorer per property. Deterministic checks for structure and computable facts; semantic similarity for paraphrase; a rubric-driven LLM judge for open prose; humans for calibration and decisive edge cases.
- 5Calibrate the scorer. Hand-label a sample and measure agreement. Inspect disagreements individually, because a judge that agrees 70% of the time is only useful if you know which 30% it gets wrong.
- 6Size the sample. A pass rate from $n$ cases has standard error $\sqrt{p(1-p)/n}$. At $p = 0.9$ and $n = 100$ that is 0.03, so a 95% interval spans roughly ±6 points. A three-point "win" from that run is noise.
Step 1 of 6
Start from the decision. Is the eval there to block a release, choose between prompts, set a routing threshold, or detect drift? The decision fixes the metric, the sample size and the acceptable error. An eval that informs nothing becomes a dashboard nobody reads.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Start from the decision. Is the eval there to block a release, choose between prompts, se… 2. Decompose the system. For RAG, retrieval recall, context precision and answer faithfulnes… 3. Define cases with grading criteria. A case is an input plus the rule that decides whether… 4. Pick a scorer per property. Deterministic checks for structure and computable facts; sema… 5. Calibrate the scorer. Hand-label a sample and measure agreement. Inspect disagreements in… 6. Size the sample. A pass rate from $n$ cases has standard error $\sqrt{p(1-p)/n}$. At $p =…
Text version
A frozen dataset is run against the candidate; outputs are graded with rules and judges; failures are analysed by slice; the release gate ships or blocks the change.
- Start from the decision. Is the eval there to block a release, choose between prompts, set a routing threshold, or detect drift? The decision fixes the metric, the sample size and the acceptable error. An eval that informs nothing becomes a dashboard nobody reads.
- Decompose the system. For RAG, retrieval recall, context precision and answer faithfulness are separate measurements. For an agent, task success, tool-call validity and trajectory length are separate. End-to-end scores gate releases; stage scores explain failures.
- Define cases with grading criteria. A case is an input plus the rule that decides whether an output is acceptable: exact match, schema check, rubric or reference answer. Datasets and golden sets covers building one that represents traffic.
- Pick a scorer per property. Deterministic checks for structure and computable facts; semantic similarity for paraphrase; a rubric-driven LLM judge for open prose; humans for calibration and decisive edge cases.
- Calibrate the scorer. Hand-label a sample and measure agreement. Inspect disagreements individually, because a judge that agrees 70% of the time is only useful if you know which 30% it gets wrong.
- Size the sample. A pass rate from cases has standard error . At and that is 0.03, so a 95% interval spans roughly ±6 points. A three-point "win" from that run is noise.
- Automate and gate. Run on every prompt, model, retrieval or dependency change. Store per-case outputs and scores, not just the aggregate, so a regression has a name and a diff.
- Close the loop online. Offline sets are frozen samples. Production signals — thumbs, edits, escalation, task completion — tell you when the sample stopped representing the population.
A minimal harness
The design choice that matters is recording every case with the scorer version attached.
from dataclasses import dataclass, field
from math import sqrt
from typing import Callable
@dataclass
class Case:
id: str
input: str
check: Callable[[str], bool]
@dataclass
class Run:
scorer_version: str
results: list[dict] = field(default_factory=list)
def add(self, case: Case, output: str) -> None:
self.results.append({"id": case.id, "passed": case.check(output), "output": output})
def summary(self) -> dict:
n = len(self.results)
if n == 0:
raise ValueError("no results to summarise")
p = sum(r["passed"] for r in self.results) / n
se = sqrt(p * (1 - p) / n)
return {"n": n, "pass_rate": round(p, 4),
"ci95": (round(p - 1.96 * se, 4), round(p + 1.96 * se, 4)),
"scorer_version": self.scorer_version}Nothing here calls a model, deliberately: the harness stays testable with fixtures and the model call remains one replaceable function at the edge.
Trade-offs
| Approach | Good for | Cost | Fails when |
|---|---|---|---|
| Manual spot checks | Early discovery, taste | No denominator, does not scale | You need to detect a regression |
| Deterministic assertions | Schema, citations, numeric facts | Cheap and stable | Output is free-form and wording varies |
| LLM judge | Open prose, rubrics, pairwise comparison | Token cost, bias, version drift | The judge is uncalibrated or the rubric is vague |
| Human review | Ground truth, ambiguous cases | Slow and expensive | Volume is too low to catch rare failures |
| Online metrics only | Real users, real distribution | Delayed and confounded by traffic mix | Traffic is low or the change is gradual |
Good for
Early discovery, taste
Cost
No denominator, does not scale
Fails when
You need to detect a regression
- You need to detect a regression
- Output is free-form and wording varies
- The judge is uncalibrated or the rubric is vague
- Volume is too low to catch rare failures
- Traffic is low or the change is gradual
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Manual spot checks is good for Early discovery, taste, costs No denominator, does not scale, and fails when You need to detect a regression. Deterministic assertions is good for Schema, citations, numeric facts, costs Cheap and stable, and fails when Output is free-form and wording varies. LLM judge is good for Open prose, rubrics, pairwise comparison, costs Token cost, bias, version drift, and fails when The judge is uncalibrated or the rubric is vague. Human review is good for Ground truth, ambiguous cases, costs Slow and expensive, and fails when Volume is too low to catch rare failures. Online metrics only is good for Real users, real distribution, costs Delayed and confounded by traffic mix, and fails when Traffic is low or the change is gradual
Failure modes
The set becomes the target. Prompts are tuned against the same 100 cases until the score is excellent and the product is not. Debug by holding out a rotating sample and tracking the gap between development and holdout scores.
The metric does not match the outcome. Faithfulness rises while users still find answers unhelpful, because the system became cautious. Debug by correlating scores with human ratings or escalations on the same cases.
The sample is too small to decide. A two-point difference is declared a win at n=40. Debug by computing the interval before looking at the direction of the change.
The harness is broken. A scorer crashes on empty output, or a judge returns unparseable text counted as a failure. Debug with golden cases whose verdicts are known, and alert on scorer error rate separately from model failure rate.
In production
Offline suites run in CI on every change to prompts, models, retrieval or tools, and should be cheap enough to block a merge: a few hundred deterministic cases plus a sampled judge run for prose. Store each run with the git SHA, prompt hash, model identifier and scorer version, because a score without provenance cannot be compared later.
Online, sample requests for asynchronous scoring rather than scoring everything. Track the same properties over time and segment by intent, since aggregate quality can stay flat while one segment collapses. Alert on input-distribution shifts as well as score drops; many "model regressions" are changes in what users ask.
Common misconceptions
People think: Evaluation means running a public benchmark and reporting the score.
Actually: A public benchmark measures a model on someone else's distribution. Your eval measures your system on your traffic with your grading rules. Benchmarks help with model selection and are nearly useless as a release gate.
People think: Once the eval suite exists, quality is solved.
Actually: The suite is a sample of proxies. It needs maintenance: refreshed cases, recalibrated judges, and metrics that still correlate with what users care about.
People think: An LLM judge removes the need for human labelling.
Actually: A judge scales human judgement; it does not replace the labels that prove the judge is right.
Interview angle
The common question is "how do you know your RAG system got better?" A weak answer cites one example or one benchmark. A strong answer names the decision, separates retrieval from generation metrics, describes the case set and its provenance, states how the judge was calibrated, and reports uncertainty — then says what would make the team revert. See the evaluation question bank and eval-driven development.
Sources
- Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — why eval comparisons need intervals and paired designs.
- Liang et al., Holistic Evaluation of Language Models — multi-metric evaluation across scenarios.
- Ribeiro et al., Beyond Accuracy: Behavioral Testing of NLP Models with CheckList — behaviour-focused test design.
- OpenAI, Evals repository — a reference implementation of case-based eval runs.
Related
- Datasets & golden sets — building the case set this page assumes.
- LLM-as-judge — the scorer most teams reach for, and its calibration.
- Regression — turning evals into a release gate.
- Error analysis — finding the failures worth measuring.
- Observability — the online half of the loop.