Why it matters
A product needs to prevent some inputs and outputs, and the model's own refusal behaviour is neither complete nor consistent. Relying on it alone produces both misses and false positives, and gives no audit trail. Moderation adds an explicit, testable policy layer, but it too is a classifier with a threshold, so the engineering problem is calibrating two imperfect filters and deciding what happens when they disagree.
Mental model
Moderation is a door policy at a venue: a check at the entrance, independent of what happens inside. The metaphor breaks because the same content can pass the door and still be harmful in context, and the bouncer cannot see the conversation.
How it works
- 1Write the policy as categories with examples of allowed and disallowed content, including context-dependent cases.
- 2Place moderation at input and output separately, since the same text can be acceptable as a question and not as an answer.
- 3Set thresholds per category from a labelled sample, and record the trade-off between misses and over-blocks.
- 4Define behaviour on a block: refuse, redirect, redact or escalate, and keep the message consistent with the policy.
- 5Handle model refusals as a separate signal, with their own logging and a way to distinguish policy refusal from capability failure.
- 6Review blocked samples on a schedule, because both policies and user behaviour drift.
Step 1 of 6
Write the policy as categories with examples of allowed and disallowed content, including context-dependent cases.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write the policy as categories with examples of allowed and disallowed content, including… 2. Place moderation at input and output separately, since the same text can be acceptable as… 3. Set thresholds per category from a labelled sample, and record the trade-off between miss… 4. Define behaviour on a block: refuse, redirect, redact or escalate, and keep the message c… 5. Handle model refusals as a separate signal, with their own logging and a way to distingui… 6. Review blocked samples on a schedule, because both policies and user behaviour drift.
- Write the policy as categories with examples of allowed and disallowed content, including context-dependent cases.
- Place moderation at input and output separately, since the same text can be acceptable as a question and not as an answer.
- Set thresholds per category from a labelled sample, and record the trade-off between misses and over-blocks.
- Define behaviour on a block: refuse, redirect, redact or escalate, and keep the message consistent with the policy.
- Handle model refusals as a separate signal, with their own logging and a way to distinguish policy refusal from capability failure.
- Review blocked samples on a schedule, because both policies and user behaviour drift.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Model refusal only | No extra service, natural language | Inconsistent, no audit, easy to talk around | The domain is regulated or high risk |
| External moderation, strict threshold | Few harmful outputs | Over-blocking legitimate requests | Users are technical and vocabulary overlaps with abuse |
| External moderation, loose threshold | Few false positives | Harmful content reaches users | The product is public and adversarial |
| Human review queue | Best judgement on edge cases | Latency and cost per item | Volume is high and responses are interactive |
Good for
No extra service, natural language
Cost
Inconsistent, no audit, easy to talk around
Fails when
The domain is regulated or high risk
- The domain is regulated or high risk
- Users are technical and vocabulary overlaps with abuse
- The product is public and adversarial
- Volume is high and responses are interactive
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Model refusal only is good for No extra service, natural language, costs Inconsistent, no audit, easy to talk around, and fails when The domain is regulated or high risk. External moderation, strict threshold is good for Few harmful outputs, costs Over-blocking legitimate requests, and fails when Users are technical and vocabulary overlaps with abuse. External moderation, loose threshold is good for Few false positives, costs Harmful content reaches users, and fails when The product is public and adversarial. Human review queue is good for Best judgement on edge cases, costs Latency and cost per item, and fails when Volume is high and responses are interactive
Failure modes
- Over-blocking. Legitimate requests about medical or security topics are refused and users leave. Debug: sample blocked inputs weekly and label them allowed or disallowed to estimate the false-positive rate.
- Refusal with no signal. The user sees a generic decline and support cannot tell whether policy or capability caused it. Debug: log a reason code per layer and expose it to support tooling.
- Output-only moderation. Harmful instructions are accepted as input and only the answer is filtered, producing inconsistent behaviour. Debug: compare block reasons at input and output for the same session.
In production
Track block rate per category and the share of blocks later overturned by human review. Rehearse a policy change by replaying a labelled sample before enabling it.
Common misconceptions
People think: A safety-trained model will not produce harmful output.
Actually: Refusal behaviour is probabilistic and prompt-dependent; it is a mitigation, not a guarantee.
People think: Moderation is censorship with no cost.
Actually: Every threshold trades missed harm against blocked legitimate use, and the cost is paid by real users.
Interview angle
Separate the model's refusal behaviour from the product's policy layer, then talk about thresholds and the false-positive budget. Strong answers describe logging, review and how the two layers interact. Weak answers assume a safety-trained model is sufficient and have no measurement for over-blocking.
Sources
- OpenAI, Moderation guide — a classification endpoint with per-category scores.
- Inan et al., Llama Guard — an input-output safety classifier and its taxonomy.
- Bai et al., Constitutional AI — training models to critique and revise outputs against principles.