Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 25 of 28

Regression

0 of 28 complete

Why it matters

Prompts, models and retrieval configuration change constantly, and each change can improve one behaviour while quietly breaking another. Aggregate scores hide that trade because gains and losses cancel. Regression evaluation stores a baseline with the exact configuration and compares per case, so a drop is visible, attributable and arguable before it reaches users. A gate that blocks on noise is as harmful as one that misses a real drop, so repeated runs and per-slice intervals belong in the design rather than in a later investigation.

Mental model

It is a canary in a mine. The metaphor breaks because the canary is also a bird you replaced last week, so you must compare like with like, and a dead canary from a bad comparison says nothing about the air.

How it works

Mechanism, step by stepInteractive
  1. 1Freeze a baseline run with the model version, prompt, retrieval config and decoding settings.
  2. 2Store per-case outputs rather than only the aggregate so a diff is possible.
  3. 3Re-run the same cases on the candidate with identical settings and compare case by case.
  4. 4Separate deterministic failures from stochastic flips by repeating sampled cases.
  5. 5Block on hard failures and on credible drops in the primary slice.
  6. 6Retire cases that pass in every configuration and add cases for each new failure.

Step 1 of 6

Freeze a baseline run with the model version, prompt, retrieval config and decoding settings.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Freeze a baseline run with the model version, prompt, retrieval config and decoding setti… 2. Store per-case outputs rather than only the aggregate so a diff is possible. 3. Re-run the same cases on the candidate with identical settings and compare case by case. 4. Separate deterministic failures from stochastic flips by repeating sampled cases. 5. Block on hard failures and on credible drops in the primary slice. 6. Retire cases that pass in every configuration and add cases for each new failure.

The mean cancels both tailsInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. The per-case score change from the stored baseline, in five-point bins, with the cumulative share below each change. Nearly half the suite gained more than five points and a fifth lost more than ten, while the average stays near zero because the two blocks cancel. The mixture is stated in the footnote and every point is generated.

What to notice. The suite scores +0.2 points against its baseline, which reads as noise. The distribution says otherwise: 47.9% of cases gained more than five points and 19.9% lost more than ten, so the release improved most of the product and broke a block of it. A gate on the mean ships the broken block; a gate on the tail costs the improvement unless the block is fixed first.

Text version

Per-case score change from minus 45 to plus 20 points, in five-point bins, with the share at or below each change below it. The density has a block centred near plus 6.5 that peaks at 33% per bin, and a smaller block near minus 20 at 7.5%; a vertical line marks a gate at minus 10. The cumulative curve reaches 12% below minus 20 and 19.9% below the gate. A slider reads both the share in a bin and the share at or below it; a marker on the tail reports that 12% lose at least 20 points, and a panel reports the +0.2-point mean, the 19.9% below the gate and the 47.9% gaining over five.

  1. Freeze a baseline run with the model version, prompt, retrieval config and decoding settings.
  2. Store per-case outputs rather than only the aggregate so a diff is possible.
  3. Re-run the same cases on the candidate with identical settings and compare case by case.
  4. Separate deterministic failures from stochastic flips by repeating sampled cases.
  5. Block on hard failures and on credible drops in the primary slice.
  6. Retire cases that pass in every configuration and add cases for each new failure.

Trade-offs

ChoiceGood forCostFails when
Strict per-case gateCatches small regressionsFlaky blocks, maintenanceOutputs are stochastic
Aggregate thresholdFewer false alarmsHides localised dropsQuality is slice-dependent
Full suite on every commitFast feedbackRuntime costSuite is large
Nightly suiteCheapSlow detectionChange velocity is high
Trade-off mapInteractive

Good for

Catches small regressions

Cost

Flaky blocks, maintenance

Fails when

Outputs are stochastic

  1. Outputs are stochastic
  2. Quality is slice-dependent
  3. Suite is large
  4. Change velocity is high

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Strict per-case gate is good for Catches small regressions, costs Flaky blocks, maintenance, and fails when Outputs are stochastic. Aggregate threshold is good for Fewer false alarms, costs Hides localised drops, and fails when Quality is slice-dependent. Full suite on every commit is good for Fast feedback, costs Runtime cost, and fails when Suite is large. Nightly suite is good for Cheap, costs Slow detection, and fails when Change velocity is high

Failure modes

  • Baseline drift. The baseline was recorded with an older prompt and every comparison is invalid. Debug: print the config hash stored with the baseline and with the candidate.
  • Flaky gate. A case flips on sampling and blocks unrelated changes. Debug: measure the case's pass rate over repeated runs before including it in the gate.
  • Silent slice loss. The aggregate holds while one category collapses. Debug: compare per-slice deltas and flag any slice with a drop beyond its interval.

In production

Store baselines with config hashes and alert on suite runtime and flake rate. Re-baseline deliberately after an accepted quality trade-off.

Common misconceptions

People think: A stable aggregate means nothing regressed.

Actually: Offsetting gains and losses cancel in the mean.

People think: Every score drop is a regression.

Actually: Some drops are sampling noise, so repeat before blocking.

Interview angle

Store the baseline configuration, compare per case, separate noise from signal and prune stale cases. A strong answer describes what the gate blocks on. Comparing aggregate scores across mismatched configs, with no localisation, is the weak version.

Sources