Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 1 of 28

Why evaluation

0 of 28 complete

Why it matters

A language model does not return the same output twice, and quality is not one property. An answer can be correct but unreadable, grounded but incomplete, polite but wrong about a number. Change a prompt, a retrieval parameter or a model version and the system moves in several dimensions at once.

Evaluation converts that judgement into a comparison with a denominator. "The new prompt seemed better in five examples" is a demo. "Pass rate on 240 labelled cases moved from 0.81 to 0.86, with an interval that excludes zero, and retrieval metrics were unchanged" is a decision.

The cost asymmetry is the argument. A bad prompt caught offline costs minutes. The same prompt caught by a customer costs trust and an incident review.

Mental model

Think of an aircraft instrument panel, not a single grade. Each gauge measures one quantity, is reliable in a range, and is cross-checked against another before you act. Airspeed alone will not tell you that you are stalling.

The analogy breaks in one important way. A physical gauge is calibrated by its manufacturer. Every scorer you add — a regex, a similarity threshold, a model judge — arrives uncalibrated. Until you compare it with human labels on your own data, precision is not evidence of accuracy.

How it works

Mechanism, step by stepInteractive
  1. 1Start from the decision. Is the eval there to block a release, choose between prompts, set a routing threshold, or detect drift? The decision fixes the metric, the sample size and the acceptable error. An eval that informs nothing becomes a dashboard nobody reads.
  2. 2Decompose the system. For RAG, retrieval recall, context precision and answer faithfulness are separate measurements. For an agent, task success, tool-call validity and trajectory length are separate. End-to-end scores gate releases; stage scores explain failures.
  3. 3Define cases with grading criteria. A case is an input plus the rule that decides whether an output is acceptable: exact match, schema check, rubric or reference answer. Datasets and golden sets covers building one that represents traffic.
  4. 4Pick a scorer per property. Deterministic checks for structure and computable facts; semantic similarity for paraphrase; a rubric-driven LLM judge for open prose; humans for calibration and decisive edge cases.
  5. 5Calibrate the scorer. Hand-label a sample and measure agreement. Inspect disagreements individually, because a judge that agrees 70% of the time is only useful if you know which 30% it gets wrong.
  6. 6Size the sample. A pass rate from $n$ cases has standard error $\sqrt{p(1-p)/n}$. At $p = 0.9$ and $n = 100$ that is 0.03, so a 95% interval spans roughly ±6 points. A three-point "win" from that run is noise.

Step 1 of 6

Start from the decision. Is the eval there to block a release, choose between prompts, set a routing threshold, or detect drift? The decision fixes the metric, the sample size and the acceptable error. An eval that informs nothing becomes a dashboard nobody reads.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Start from the decision. Is the eval there to block a release, choose between prompts, se… 2. Decompose the system. For RAG, retrieval recall, context precision and answer faithfulnes… 3. Define cases with grading criteria. A case is an input plus the rule that decides whether… 4. Pick a scorer per property. Deterministic checks for structure and computable facts; sema… 5. Calibrate the scorer. Hand-label a sample and measure agreement. Inspect disagreements in… 6. Size the sample. A pass rate from $n$ cases has standard error $\sqrt{p(1-p)/n}$. At $p =…

passDatasetRunGradeAnalyseGateRelease
Step 1/5 · Frozen dataset and golden set
Figure. Evaluation is a loop: a frozen dataset, a candidate run, grading, slice analysis, and a release gate.
Text version

A frozen dataset is run against the candidate; outputs are graded with rules and judges; failures are analysed by slice; the release gate ships or blocks the change.

  1. Start from the decision. Is the eval there to block a release, choose between prompts, set a routing threshold, or detect drift? The decision fixes the metric, the sample size and the acceptable error. An eval that informs nothing becomes a dashboard nobody reads.
  2. Decompose the system. For RAG, retrieval recall, context precision and answer faithfulness are separate measurements. For an agent, task success, tool-call validity and trajectory length are separate. End-to-end scores gate releases; stage scores explain failures.
  3. Define cases with grading criteria. A case is an input plus the rule that decides whether an output is acceptable: exact match, schema check, rubric or reference answer. Datasets and golden sets covers building one that represents traffic.
  4. Pick a scorer per property. Deterministic checks for structure and computable facts; semantic similarity for paraphrase; a rubric-driven LLM judge for open prose; humans for calibration and decisive edge cases.
  5. Calibrate the scorer. Hand-label a sample and measure agreement. Inspect disagreements individually, because a judge that agrees 70% of the time is only useful if you know which 30% it gets wrong.
  6. Size the sample. A pass rate from nn cases has standard error p(1−p)/n\sqrt{p(1-p)/n}. At p=0.9p = 0.9 and n=100n = 100 that is 0.03, so a 95% interval spans roughly ±6 points. A three-point "win" from that run is noise.
  7. Automate and gate. Run on every prompt, model, retrieval or dependency change. Store per-case outputs and scores, not just the aggregate, so a regression has a name and a diff.
  8. Close the loop online. Offline sets are frozen samples. Production signals — thumbs, edits, escalation, task completion — tell you when the sample stopped representing the population.

A minimal harness

The design choice that matters is recording every case with the scorer version attached.

from dataclasses import dataclass, field
from math import sqrt
from typing import Callable
 
@dataclass
class Case:
    id: str
    input: str
    check: Callable[[str], bool]
 
@dataclass
class Run:
    scorer_version: str
    results: list[dict] = field(default_factory=list)
 
    def add(self, case: Case, output: str) -> None:
        self.results.append({"id": case.id, "passed": case.check(output), "output": output})
 
    def summary(self) -> dict:
        n = len(self.results)
        if n == 0:
            raise ValueError("no results to summarise")
        p = sum(r["passed"] for r in self.results) / n
        se = sqrt(p * (1 - p) / n)
        return {"n": n, "pass_rate": round(p, 4),
                "ci95": (round(p - 1.96 * se, 4), round(p + 1.96 * se, 4)),
                "scorer_version": self.scorer_version}

Nothing here calls a model, deliberately: the harness stays testable with fixtures and the model call remains one replaceable function at the edge.

Trade-offs

ApproachGood forCostFails when
Manual spot checksEarly discovery, tasteNo denominator, does not scaleYou need to detect a regression
Deterministic assertionsSchema, citations, numeric factsCheap and stableOutput is free-form and wording varies
LLM judgeOpen prose, rubrics, pairwise comparisonToken cost, bias, version driftThe judge is uncalibrated or the rubric is vague
Human reviewGround truth, ambiguous casesSlow and expensiveVolume is too low to catch rare failures
Online metrics onlyReal users, real distributionDelayed and confounded by traffic mixTraffic is low or the change is gradual
Trade-off mapInteractive

Good for

Early discovery, taste

Cost

No denominator, does not scale

Fails when

You need to detect a regression

  1. You need to detect a regression
  2. Output is free-form and wording varies
  3. The judge is uncalibrated or the rubric is vague
  4. Volume is too low to catch rare failures
  5. Traffic is low or the change is gradual

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Manual spot checks is good for Early discovery, taste, costs No denominator, does not scale, and fails when You need to detect a regression. Deterministic assertions is good for Schema, citations, numeric facts, costs Cheap and stable, and fails when Output is free-form and wording varies. LLM judge is good for Open prose, rubrics, pairwise comparison, costs Token cost, bias, version drift, and fails when The judge is uncalibrated or the rubric is vague. Human review is good for Ground truth, ambiguous cases, costs Slow and expensive, and fails when Volume is too low to catch rare failures. Online metrics only is good for Real users, real distribution, costs Delayed and confounded by traffic mix, and fails when Traffic is low or the change is gradual

Failure modes

The set becomes the target. Prompts are tuned against the same 100 cases until the score is excellent and the product is not. Debug by holding out a rotating sample and tracking the gap between development and holdout scores.

The metric does not match the outcome. Faithfulness rises while users still find answers unhelpful, because the system became cautious. Debug by correlating scores with human ratings or escalations on the same cases.

The sample is too small to decide. A two-point difference is declared a win at n=40. Debug by computing the interval before looking at the direction of the change.

The harness is broken. A scorer crashes on empty output, or a judge returns unparseable text counted as a failure. Debug with golden cases whose verdicts are known, and alert on scorer error rate separately from model failure rate.

In production

Offline suites run in CI on every change to prompts, models, retrieval or tools, and should be cheap enough to block a merge: a few hundred deterministic cases plus a sampled judge run for prose. Store each run with the git SHA, prompt hash, model identifier and scorer version, because a score without provenance cannot be compared later.

Online, sample requests for asynchronous scoring rather than scoring everything. Track the same properties over time and segment by intent, since aggregate quality can stay flat while one segment collapses. Alert on input-distribution shifts as well as score drops; many "model regressions" are changes in what users ask.

Common misconceptions

People think: Evaluation means running a public benchmark and reporting the score.

Actually: A public benchmark measures a model on someone else's distribution. Your eval measures your system on your traffic with your grading rules. Benchmarks help with model selection and are nearly useless as a release gate.

People think: Once the eval suite exists, quality is solved.

Actually: The suite is a sample of proxies. It needs maintenance: refreshed cases, recalibrated judges, and metrics that still correlate with what users care about.

People think: An LLM judge removes the need for human labelling.

Actually: A judge scales human judgement; it does not replace the labels that prove the judge is right.

Interview angle

The common question is "how do you know your RAG system got better?" A weak answer cites one example or one benchmark. A strong answer names the decision, separates retrieval from generation metrics, describes the case set and its provenance, states how the judge was calibrated, and reports uncertainty — then says what would make the team revert. See the evaluation question bank and eval-driven development.

Sources