Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 4 of 28

Rule metrics

0 of 28 complete

Why it matters

Some properties of an output are binary and cheap to check: the JSON parses, the required key exists, the citation id resolves, the answer is one of four labels. A model judge is unnecessary and noisy for those. Rule metrics turn them into gates that run on every request, but the same determinism makes them blind to meaning, so they can pass an answer that is perfectly formatted and completely wrong.

Mental model

It is a spell-checker for a contract: it catches typos, not a bad clause. The metaphor breaks because a flawlessly formatted response can still be false, and the checker has nothing to say about meaning, so a green gate is not a verdict.

How it works

The last wrong answers cost the mostInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Two score distributions behind one threshold: the share of wrong outputs rejected and the share of valid outputs rejected as the rule tightens. Both curves rise, but the valid-output curve turns up steeply once the rule is already catching most wrong answers.

What to notice. At a 0.60 threshold the rule rejects 88.5% of wrong outputs and 6.7% of valid ones. Moving to 0.70 catches another 8.4 points of wrong answers but rejects 25.2% of valid ones: 221 valid answers lost for every 100 extra wrong ones caught, against 26 per 100 caught overall. Tightening further to 0.80 rejects more than half of all valid outputs.

Text version

Two curves against a rule threshold from 0.20 to 0.90. The wrong-outputs-caught curve rises from 7% at 0.20 to 70.3% at 0.50, 88.5% at 0.60 and 96.9% at 0.70. The valid-outputs-rejected curve stays near zero until 0.50, then rises to 6.7% at 0.60, 25.2% at 0.70 and 56.6% at 0.80. Three buttons set the threshold to 0.50, 0.60 or 0.70; markers on both curves report the shares at the chosen threshold and a panel reports 221 valid answers lost per 100 extra wrong ones caught between 0.60 and 0.70.

  1. List the properties that are genuinely binary, such as parse success, required keys and resolvable references.
  2. Write each rule against the output contract and test it with one passing and one failing example.
  3. Separate hard gates from soft warnings, and fail the run only on the hard set.
  4. Normalise whitespace and casing before matching so formatting does not decide the score.
  5. Version the rules with the prompt, since an output format change invalidates them.
  6. Pair every rule with a sampled human or judge check, because rules miss semantic failure.

Trade-offs

ChoiceGood forCostFails when
Schema validationExact, fast, debuggableOnly structureOutput is free-form prose
Exact matchUnambiguousPunishes valid paraphrasesSeveral correct answers exist
Regex extractionWorks on semi-structured textBrittle to format driftModel output varies in phrasing
Rule plus judgeCoverage and determinismTwo systems to maintainRules already cover the case
Trade-off mapInteractive

Good for

Exact, fast, debuggable

Cost

Only structure

Fails when

Output is free-form prose

  1. Output is free-form prose
  2. Several correct answers exist
  3. Model output varies in phrasing
  4. Rules already cover the case

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Schema validation is good for Exact, fast, debuggable, costs Only structure, and fails when Output is free-form prose. Exact match is good for Unambiguous, costs Punishes valid paraphrases, and fails when Several correct answers exist. Regex extraction is good for Works on semi-structured text, costs Brittle to format drift, and fails when Model output varies in phrasing. Rule plus judge is good for Coverage and determinism, costs Two systems to maintain, and fails when Rules already cover the case

Failure modes

  • Over-strict match. A correct answer fails because of a trailing period. Debug: print the normalised expected and actual strings for one failing case.
  • Rule drift. A prompt change alters the output format and every case fails the schema rule. Debug: run the rule suite against a known-good output from the previous prompt version.
  • Proxy gaming. The agent satisfies the rule without solving the task. Debug: add a held-out semantic check and compare pass rates between the two.

In production

Run rules on every request as a cheap gate and log which rule fired. Alert on a spike in one rule, which usually means an output format changed.

Common misconceptions

People think: Deterministic metrics are objective.

Actually: They are reproducible; whether they measure the right property is a separate question.

People think: If the rules pass, quality is fine.

Actually: Rules see structure and keywords, not whether the answer is useful or true.

Interview angle

Use rules for binary properties, version them with the output contract and keep a semantic check alongside. A strong answer gives an example of a rule that passes a wrong answer. Treating exact match as general quality, with no normalisation or drift, is the weak version.

Sources