Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 23 of 28

Refusals & moderation

0 of 28 complete

Why it matters

A product needs to prevent some inputs and outputs, and the model's own refusal behaviour is neither complete nor consistent. Relying on it alone produces both misses and false positives, and gives no audit trail. Moderation adds an explicit, testable policy layer, but it too is a classifier with a threshold, so the engineering problem is calibrating two imperfect filters and deciding what happens when they disagree.

Mental model

Moderation is a door policy at a venue: a check at the entrance, independent of what happens inside. The metaphor breaks because the same content can pass the door and still be harmful in context, and the bouncer cannot see the conversation.

How it works

Mechanism, step by stepInteractive
  1. 1Write the policy as categories with examples of allowed and disallowed content, including context-dependent cases.
  2. 2Place moderation at input and output separately, since the same text can be acceptable as a question and not as an answer.
  3. 3Set thresholds per category from a labelled sample, and record the trade-off between misses and over-blocks.
  4. 4Define behaviour on a block: refuse, redirect, redact or escalate, and keep the message consistent with the policy.
  5. 5Handle model refusals as a separate signal, with their own logging and a way to distinguish policy refusal from capability failure.
  6. 6Review blocked samples on a schedule, because both policies and user behaviour drift.

Step 1 of 6

Write the policy as categories with examples of allowed and disallowed content, including context-dependent cases.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the policy as categories with examples of allowed and disallowed content, including… 2. Place moderation at input and output separately, since the same text can be acceptable as… 3. Set thresholds per category from a labelled sample, and record the trade-off between miss… 4. Define behaviour on a block: refuse, redirect, redact or escalate, and keep the message c… 5. Handle model refusals as a separate signal, with their own logging and a way to distingui… 6. Review blocked samples on a schedule, because both policies and user behaviour drift.

  1. Write the policy as categories with examples of allowed and disallowed content, including context-dependent cases.
  2. Place moderation at input and output separately, since the same text can be acceptable as a question and not as an answer.
  3. Set thresholds per category from a labelled sample, and record the trade-off between misses and over-blocks.
  4. Define behaviour on a block: refuse, redirect, redact or escalate, and keep the message consistent with the policy.
  5. Handle model refusals as a separate signal, with their own logging and a way to distinguish policy refusal from capability failure.
  6. Review blocked samples on a schedule, because both policies and user behaviour drift.

Trade-offs

ChoiceGood forCostFails when
Model refusal onlyNo extra service, natural languageInconsistent, no audit, easy to talk aroundThe domain is regulated or high risk
External moderation, strict thresholdFew harmful outputsOver-blocking legitimate requestsUsers are technical and vocabulary overlaps with abuse
External moderation, loose thresholdFew false positivesHarmful content reaches usersThe product is public and adversarial
Human review queueBest judgement on edge casesLatency and cost per itemVolume is high and responses are interactive
Trade-off mapInteractive

Good for

No extra service, natural language

Cost

Inconsistent, no audit, easy to talk around

Fails when

The domain is regulated or high risk

  1. The domain is regulated or high risk
  2. Users are technical and vocabulary overlaps with abuse
  3. The product is public and adversarial
  4. Volume is high and responses are interactive

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Model refusal only is good for No extra service, natural language, costs Inconsistent, no audit, easy to talk around, and fails when The domain is regulated or high risk. External moderation, strict threshold is good for Few harmful outputs, costs Over-blocking legitimate requests, and fails when Users are technical and vocabulary overlaps with abuse. External moderation, loose threshold is good for Few false positives, costs Harmful content reaches users, and fails when The product is public and adversarial. Human review queue is good for Best judgement on edge cases, costs Latency and cost per item, and fails when Volume is high and responses are interactive

Failure modes

  • Over-blocking. Legitimate requests about medical or security topics are refused and users leave. Debug: sample blocked inputs weekly and label them allowed or disallowed to estimate the false-positive rate.
  • Refusal with no signal. The user sees a generic decline and support cannot tell whether policy or capability caused it. Debug: log a reason code per layer and expose it to support tooling.
  • Output-only moderation. Harmful instructions are accepted as input and only the answer is filtered, producing inconsistent behaviour. Debug: compare block reasons at input and output for the same session.

In production

Track block rate per category and the share of blocks later overturned by human review. Rehearse a policy change by replaying a labelled sample before enabling it.

Common misconceptions

People think: A safety-trained model will not produce harmful output.

Actually: Refusal behaviour is probabilistic and prompt-dependent; it is a mitigation, not a guarantee.

People think: Moderation is censorship with no cost.

Actually: Every threshold trades missed harm against blocked legitimate use, and the cost is paid by real users.

Interview angle

Separate the model's refusal behaviour from the product's policy layer, then talk about thresholds and the false-positive budget. Strong answers describe logging, review and how the two layers interact. Weak answers assume a safety-trained model is sufficient and have no measurement for over-blocking.

Sources