Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 21 of 28

Safety evaluation

0 of 28 complete

Why it matters

A safety policy describes many distinct harms, and a system can be robust on one category while failing another. Measuring only harmful compliance also rewards a system that refuses everything, which is a product failure. For agents the surface widens from words to actions, so evaluation has to cover forbidden tools, paths and network destinations in addition to conversational refusal.

Mental model

It is a building's fire drill: exits, alarms and assembly points. The metaphor breaks because the fire can be a stranger talking the guard into unlocking a door, which no drill scenario covers and no alarm reports, and the guard may be the one who was talked to.

How it works

Mechanism, step by stepInteractive
  1. 1Write the policy as testable categories with examples of prohibited and allowed behaviour.
  2. 2Build prompts per category, including benign near-misses that should be answered.
  3. 3Measure harmful compliance and over-refusal on the benign set together.
  4. 4Add action-level tests for agents: forbidden tools, paths and network destinations.
  5. 5Have humans adjudicate a sample, since harm classifiers are themselves imperfect.
  6. 6Re-run after policy, model or tool changes and record the version of each.

Step 1 of 6

Write the policy as testable categories with examples of prohibited and allowed behaviour.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the policy as testable categories with examples of prohibited and allowed behaviour. 2. Build prompts per category, including benign near-misses that should be answered. 3. Measure harmful compliance and over-refusal on the benign set together. 4. Add action-level tests for agents: forbidden tools, paths and network destinations. 5. Have humans adjudicate a sample, since harm classifiers are themselves imperfect. 6. Re-run after policy, model or tool changes and record the version of each.

  1. Write the policy as testable categories with examples of prohibited and allowed behaviour.
  2. Build prompts per category, including benign near-misses that should be answered.
  3. Measure harmful compliance and over-refusal on the benign set together.
  4. Add action-level tests for agents: forbidden tools, paths and network destinations.
  5. Have humans adjudicate a sample, since harm classifiers are themselves imperfect.
  6. Re-run after policy, model or tool changes and record the version of each.

Trade-offs

ChoiceGood forCostFails when
Classifier-scored harmCheap at scaleClassifier false positives and negativesNuanced or dual-use content
Human adjudicationContext-awareSlow and expensiveVolume is high
Refusal-rate targetSimple to trackEncourages over-refusalHelpfulness matters
Action-level sandbox testsCatches real harmComplex setupRisk is purely conversational
Trade-off mapInteractive

Good for

Cheap at scale

Cost

Classifier false positives and negatives

Fails when

Nuanced or dual-use content

  1. Nuanced or dual-use content
  2. Volume is high
  3. Helpfulness matters
  4. Risk is purely conversational

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Classifier-scored harm is good for Cheap at scale, costs Classifier false positives and negatives, and fails when Nuanced or dual-use content. Human adjudication is good for Context-aware, costs Slow and expensive, and fails when Volume is high. Refusal-rate target is good for Simple to track, costs Encourages over-refusal, and fails when Helpfulness matters. Action-level sandbox tests is good for Catches real harm, costs Complex setup, and fails when Risk is purely conversational

Failure modes

  • Over-refusal. Benign requests in sensitive domains are refused and the safety score looks good. Debug: report the refusal rate on the benign control set separately.
  • Category collapse. One harm category is untested and the aggregate hides it. Debug: print results per policy category and count cases per category.
  • Policy drift. The policy changed and the labels still encode the old one. Debug: diff the policy version against the label version stored with the set.

In production

Track refusal rates on harmful and benign sets together, and action denials per category. Review the policy diff before each eval refresh.

Common misconceptions

People think: A low harmful-compliance rate means the system is safe.

Actually: It may be refusing too much or missing a category entirely.

People think: Safety evaluation is a one-time launch gate.

Actually: Policies, models and tools change, and each change can reopen a category.

Interview angle

Treat safety as a set of categories, measure over-refusal alongside harmful compliance and extend tests to actions. A strong answer mentions policy versioning. A single refusal number, with no benign control set, is the weak response.

Sources