Why it matters
Some properties of an output are binary and cheap to check: the JSON parses, the required key exists, the citation id resolves, the answer is one of four labels. A model judge is unnecessary and noisy for those. Rule metrics turn them into gates that run on every request, but the same determinism makes them blind to meaning, so they can pass an answer that is perfectly formatted and completely wrong.
Mental model
It is a spell-checker for a contract: it catches typos, not a bad clause. The metaphor breaks because a flawlessly formatted response can still be false, and the checker has nothing to say about meaning, so a green gate is not a verdict.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Two score distributions behind one threshold: the share of wrong outputs rejected and the share of valid outputs rejected as the rule tightens. Both curves rise, but the valid-output curve turns up steeply once the rule is already catching most wrong answers.
What to notice. At a 0.60 threshold the rule rejects 88.5% of wrong outputs and 6.7% of valid ones. Moving to 0.70 catches another 8.4 points of wrong answers but rejects 25.2% of valid ones: 221 valid answers lost for every 100 extra wrong ones caught, against 26 per 100 caught overall. Tightening further to 0.80 rejects more than half of all valid outputs.
Text version
Two curves against a rule threshold from 0.20 to 0.90. The wrong-outputs-caught curve rises from 7% at 0.20 to 70.3% at 0.50, 88.5% at 0.60 and 96.9% at 0.70. The valid-outputs-rejected curve stays near zero until 0.50, then rises to 6.7% at 0.60, 25.2% at 0.70 and 56.6% at 0.80. Three buttons set the threshold to 0.50, 0.60 or 0.70; markers on both curves report the shares at the chosen threshold and a panel reports 221 valid answers lost per 100 extra wrong ones caught between 0.60 and 0.70.
- List the properties that are genuinely binary, such as parse success, required keys and resolvable references.
- Write each rule against the output contract and test it with one passing and one failing example.
- Separate hard gates from soft warnings, and fail the run only on the hard set.
- Normalise whitespace and casing before matching so formatting does not decide the score.
- Version the rules with the prompt, since an output format change invalidates them.
- Pair every rule with a sampled human or judge check, because rules miss semantic failure.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Schema validation | Exact, fast, debuggable | Only structure | Output is free-form prose |
| Exact match | Unambiguous | Punishes valid paraphrases | Several correct answers exist |
| Regex extraction | Works on semi-structured text | Brittle to format drift | Model output varies in phrasing |
| Rule plus judge | Coverage and determinism | Two systems to maintain | Rules already cover the case |
Good for
Exact, fast, debuggable
Cost
Only structure
Fails when
Output is free-form prose
- Output is free-form prose
- Several correct answers exist
- Model output varies in phrasing
- Rules already cover the case
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Schema validation is good for Exact, fast, debuggable, costs Only structure, and fails when Output is free-form prose. Exact match is good for Unambiguous, costs Punishes valid paraphrases, and fails when Several correct answers exist. Regex extraction is good for Works on semi-structured text, costs Brittle to format drift, and fails when Model output varies in phrasing. Rule plus judge is good for Coverage and determinism, costs Two systems to maintain, and fails when Rules already cover the case
Failure modes
- Over-strict match. A correct answer fails because of a trailing period. Debug: print the normalised expected and actual strings for one failing case.
- Rule drift. A prompt change alters the output format and every case fails the schema rule. Debug: run the rule suite against a known-good output from the previous prompt version.
- Proxy gaming. The agent satisfies the rule without solving the task. Debug: add a held-out semantic check and compare pass rates between the two.
In production
Run rules on every request as a cheap gate and log which rule fired. Alert on a spike in one rule, which usually means an output format changed.
Common misconceptions
People think: Deterministic metrics are objective.
Actually: They are reproducible; whether they measure the right property is a separate question.
People think: If the rules pass, quality is fine.
Actually: Rules see structure and keywords, not whether the answer is useful or true.
Interview angle
Use rules for binary properties, version them with the output contract and keep a semantic check alongside. A strong answer gives an example of a rule that passes a wrong answer. Treating exact match as general quality, with no normalisation or drift, is the weak version.
Sources
- Ribeiro et al., Beyond Accuracy with CheckList — behavioural tests expressed as explicit checks.
- JSON Schema — the specification behind structural output validation.
- Hypothesis documentation — property-based testing as a model for rule design.