Why it matters
A safety policy describes many distinct harms, and a system can be robust on one category while failing another. Measuring only harmful compliance also rewards a system that refuses everything, which is a product failure. For agents the surface widens from words to actions, so evaluation has to cover forbidden tools, paths and network destinations in addition to conversational refusal.
Mental model
It is a building's fire drill: exits, alarms and assembly points. The metaphor breaks because the fire can be a stranger talking the guard into unlocking a door, which no drill scenario covers and no alarm reports, and the guard may be the one who was talked to.
How it works
- 1Write the policy as testable categories with examples of prohibited and allowed behaviour.
- 2Build prompts per category, including benign near-misses that should be answered.
- 3Measure harmful compliance and over-refusal on the benign set together.
- 4Add action-level tests for agents: forbidden tools, paths and network destinations.
- 5Have humans adjudicate a sample, since harm classifiers are themselves imperfect.
- 6Re-run after policy, model or tool changes and record the version of each.
Step 1 of 6
Write the policy as testable categories with examples of prohibited and allowed behaviour.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write the policy as testable categories with examples of prohibited and allowed behaviour. 2. Build prompts per category, including benign near-misses that should be answered. 3. Measure harmful compliance and over-refusal on the benign set together. 4. Add action-level tests for agents: forbidden tools, paths and network destinations. 5. Have humans adjudicate a sample, since harm classifiers are themselves imperfect. 6. Re-run after policy, model or tool changes and record the version of each.
- Write the policy as testable categories with examples of prohibited and allowed behaviour.
- Build prompts per category, including benign near-misses that should be answered.
- Measure harmful compliance and over-refusal on the benign set together.
- Add action-level tests for agents: forbidden tools, paths and network destinations.
- Have humans adjudicate a sample, since harm classifiers are themselves imperfect.
- Re-run after policy, model or tool changes and record the version of each.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Classifier-scored harm | Cheap at scale | Classifier false positives and negatives | Nuanced or dual-use content |
| Human adjudication | Context-aware | Slow and expensive | Volume is high |
| Refusal-rate target | Simple to track | Encourages over-refusal | Helpfulness matters |
| Action-level sandbox tests | Catches real harm | Complex setup | Risk is purely conversational |
Good for
Cheap at scale
Cost
Classifier false positives and negatives
Fails when
Nuanced or dual-use content
- Nuanced or dual-use content
- Volume is high
- Helpfulness matters
- Risk is purely conversational
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Classifier-scored harm is good for Cheap at scale, costs Classifier false positives and negatives, and fails when Nuanced or dual-use content. Human adjudication is good for Context-aware, costs Slow and expensive, and fails when Volume is high. Refusal-rate target is good for Simple to track, costs Encourages over-refusal, and fails when Helpfulness matters. Action-level sandbox tests is good for Catches real harm, costs Complex setup, and fails when Risk is purely conversational
Failure modes
- Over-refusal. Benign requests in sensitive domains are refused and the safety score looks good. Debug: report the refusal rate on the benign control set separately.
- Category collapse. One harm category is untested and the aggregate hides it. Debug: print results per policy category and count cases per category.
- Policy drift. The policy changed and the labels still encode the old one. Debug: diff the policy version against the label version stored with the set.
In production
Track refusal rates on harmful and benign sets together, and action denials per category. Review the policy diff before each eval refresh.
Common misconceptions
People think: A low harmful-compliance rate means the system is safe.
Actually: It may be refusing too much or missing a category entirely.
People think: Safety evaluation is a one-time launch gate.
Actually: Policies, models and tools change, and each change can reopen a category.
Interview angle
Treat safety as a set of categories, measure over-refusal alongside harmful compliance and extend tests to actions. A strong answer mentions policy versioning. A single refusal number, with no benign control set, is the weak response.
Sources
- Mazeika et al., HarmBench — standardised harmful-behaviour evaluation.
- Bai et al., Constitutional AI — policy-driven harmlessness training and evaluation.
- OWASP, Top 10 for LLM Applications — harm categories relevant to deployed systems.