Why it matters
Prompt and retrieval changes are hard to reason about, so teams ship them on intuition and discover regressions later. Writing the case first turns the intended behaviour into a specification that fails before the change and passes after. The practice borrows its discipline from test-driven development, with one difference: the oracle is noisy, so a single run is not a verdict and thresholds need margins.
Mental model
It is TDD for probabilistic behaviour. The metaphor breaks because the test is stochastic, so green is a distribution rather than a fact, and a single run can pass for reasons that have nothing to do with the change under test.
How it works
- 1Take one real failure and write it as a case with an input, a check and a note on the harm.
- 2Confirm the case fails on the current system for the stated reason.
- 3Make the smallest prompt, retrieval or tool change that addresses it.
- 4Run the full suite to catch regressions, repeating runs where sampling is stochastic.
- 5Keep the change only if the target case passes and no slice regresses.
- 6Commit the case with the change so the specification grows with the product.
Step 1 of 6
Take one real failure and write it as a case with an input, a check and a note on the harm.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Take one real failure and write it as a case with an input, a check and a note on the har… 2. Confirm the case fails on the current system for the stated reason. 3. Make the smallest prompt, retrieval or tool change that addresses it. 4. Run the full suite to catch regressions, repeating runs where sampling is stochastic. 5. Keep the change only if the target case passes and no slice regresses. 6. Commit the case with the change so the specification grows with the product.
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. The same 32 eval cases bucketed by how often they pass over repeated runs. One run makes every case look binary; ten and twenty reruns move a block into the middle, where the case is neither a specification that holds nor one that fails. Counts are illustrative.
What to notice. One run reports 26 passes and six failures. Over twenty reruns the six failures stay failures, 23 cases settle above 80%, and six cases sit between 20 and 80%: they pass often enough to look green and fail often enough that a single run decides which answer you see. The target case and its verdict need a pass rate before either a keep or a revert is honest.
Text version
Horizontal bars for five pass-rate bands, counting eval cases. After twenty reruns, 23 cases pass in at least 80% of runs, two sit in 60-80%, two in 40-60%, two in 20-40% and three in 0-20%. A two-button control switches to ten reruns, where the counts are 22, 2, 3, 2 and 3, and to one run, where all 32 cases fall in the top band or the bottom band and the middle three are empty. A panel reports cases above 80%, cases between 20 and 80% and whether the verdict is a rate.
- Take one real failure and write it as a case with an input, a check and a note on the harm.
- Confirm the case fails on the current system for the stated reason.
- Make the smallest prompt, retrieval or tool change that addresses it.
- Run the full suite to catch regressions, repeating runs where sampling is stochastic.
- Keep the change only if the target case passes and no slice regresses.
- Commit the case with the change so the specification grows with the product.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Case per bug | Direct and grounded | Suite grows without structure | Volume of failures is high |
| Large suite before building | Design pressure | Slow start, speculative cases | Product direction is unclear |
| Strict no-regression gate | Stable quality | Blocks deliberate trade-offs | Trade-off is intentional |
| Manual spot checks | Fast | No memory, no comparability | Team grows |
Good for
Direct and grounded
Cost
Suite grows without structure
Fails when
Volume of failures is high
- Volume of failures is high
- Product direction is unclear
- Trade-off is intentional
- Team grows
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Case per bug is good for Direct and grounded, costs Suite grows without structure, and fails when Volume of failures is high. Large suite before building is good for Design pressure, costs Slow start, speculative cases, and fails when Product direction is unclear. Strict no-regression gate is good for Stable quality, costs Blocks deliberate trade-offs, and fails when Trade-off is intentional. Manual spot checks is good for Fast, costs No memory, no comparability, and fails when Team grows
Failure modes
- Case written to pass. The expected output is copied from current behaviour including its bug. Debug: check the case fails on the commit before the fix.
- Flaky verdict. A stochastic case flips between runs and blocks unrelated changes. Debug: run the case repeatedly and measure its pass distribution before gating on it.
- Suite rot. Cases reference prompts or endpoints that no longer exist. Debug: run the suite on main weekly and fail on errors, not only on score drops.
In production
Run the suite on every change to prompts, retrieval or models, and store per-case results. Track suite runtime and prune cases that no longer discriminate.
Common misconceptions
People think: Evals are a QA phase after building.
Actually: They are the specification, and writing them afterwards usually encodes whatever the system already does.
People think: A green suite means the feature is good.
Actually: It means the cases you wrote pass, and coverage of the behaviour is the open question.
Interview angle
Start from a real failing case, check that it fails for the right reason and keep the case with the fix. A strong answer covers stochastic margins and suite pruning. Writing evals after the feature and using them as a dashboard, rather than a gate, is the weak response.
Sources
- Hamel Husain, A Field Guide to Rapidly Improving AI Products — error analysis and eval loops in practice.
- OpenAI, Evals guide — writing and running evaluation suites.
- Shankar et al., Who Validates the Validators? — aligning automated evaluators with human judgement.