Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 14 of 26

Prompt testing

0 of 26 complete

Why it matters

Prompt changes are usually judged by reading a few outputs, and a convincing sample is easy to produce. That process cannot distinguish a real improvement from a lucky draw, and it cannot catch regressions on cases nobody thought to try. Testing replaces judgement-by-example with a repeatable measurement, which is what makes iteration fast rather than risky. It also gives reviewers something to argue about other than taste.

Mental model

Prompt testing is a driving test with a fixed route: everyone is measured on the same manoeuvres. The metaphor breaks because the route can be memorised, and a prompt tuned to the test set may fail on the road. The examiner also never sees the traffic you meet later.

How it works

The judge is a prompt tooInteractive

Scoring window · illustrative tokens

700 input+ 300 reserve= 1,000 of 1,800

800 free

Step 1/4 · Exact match

Step 1Exact match

A label comparison is code, not a call: 700 tokens of case and expected label, 800 free, and no scoring window to pay for. It only works when the answer can be compared exactly.

reserve 300700 tokens

Output reserve

300

No judge call: the 800 free tokens are the cost the exact-match check does not spend.

output reservefixedBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. Model-as-judge scoring is a second call per case: a rubric, the case input, the candidate answer and sometimes a reference share one judge window. Rubric scoring fits with 40 tokens free; adding a reference cuts 140 tokens from the answer being graded.

What to notice. Counts are authored for a 1,800-token judge window with a 300-token verdict reserve; no judge was called and no suite was run. The judge window is a composition the page names as a cost, not a number any provider publishes. Exact match is drawn as a comparison in code with no window; human review is omitted.

Text version

A 1,800-token judge window with a 300-token verdict reserve, so 1,500 tokens are available. The default first step is exact match: the case input 640 and the expected label 60, for 700 tokens and 800 free, with no scoring call at all. The second step is a rubric judge: judge instructions 90, rubric 320, case input 640 and candidate answer 410, for 1,460 tokens and 40 free. The third step adds a 180-token reference answer and runs 140 over the window, with the cells crossing the cap rule into the reserve. The fourth step keeps the reference and cuts the candidate answer from 410 to 270 tokens, filling the window exactly — the judge now grades a truncated answer. Selecting a section shows its count and role.

  1. Collect cases from real traffic, including failures, edge cases and the common path, and label the expected output or property.
  2. Choose a scoring rule per case type: exact match for labels, field-level checks for extraction, a rubric for prose.
  3. Freeze a development split for iteration and a held-out split used only for acceptance.
  4. Run every candidate prompt on both splits and record scores with the prompt and model version.
  5. Set an acceptance threshold before looking at results, including a tolerance for noise.
  6. Refresh the set on a schedule with newly observed cases, and retire stale ones.

Trade-offs

ChoiceGood forCostFails when
Exact matchObjective and cheapPunishes valid variationThe output is free-form
Model-as-judgeScales to open-ended outputJudge bias and costThe judge is the same model being tested
Human reviewBest judgementSlow and expensiveYou need per-commit feedback
Large synthetic setBroad coverageMay not resemble real trafficThe distribution is unusual
Trade-off mapInteractive

Good for

Objective and cheap

Cost

Punishes valid variation

Fails when

The output is free-form

  1. The output is free-form
  2. The judge is the same model being tested
  3. You need per-commit feedback
  4. The distribution is unusual

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Exact match is good for Objective and cheap, costs Punishes valid variation, and fails when The output is free-form. Model-as-judge is good for Scales to open-ended output, costs Judge bias and cost, and fails when The judge is the same model being tested. Human review is good for Best judgement, costs Slow and expensive, and fails when You need per-commit feedback. Large synthetic set is good for Broad coverage, costs May not resemble real traffic, and fails when The distribution is unusual

Failure modes

  • Overfitting to the test set. Scores rise while production quality falls. Debug: compare development and held-out scores, and refresh with recent traffic.
  • Scoring the wrong thing. The metric rewards format and ignores correctness. Debug: inspect a sample of high-scoring failures by hand.
  • Noisy comparison. A one-case difference is treated as an improvement. Debug: report confidence intervals or run the set multiple times with sampling.

In production

Run the suite on every prompt change and on every model upgrade, and keep results with the version ids. Sample production outputs into the set so it tracks the traffic.

Common misconceptions

People think: A prompt that works on ten examples is ready.

Actually: Ten examples show plausibility; acceptance needs a set sized for the regressions you care about.

People think: Model-as-judge removes the need for labels.

Actually: A judge needs a reference or rubric, and it inherits biases that must be measured against human labels.

Interview angle

Describe the set, the scoring rule and the split, then explain how the result changes a decision. Strong answers mention noise, overfitting and refresh. Weak answers say they try a few prompts and compare the outputs by eye.

Sources