Why it matters
A model can be persuaded, confused or simply unlucky, and the same weights will occasionally emit something the product must never ship. The engineering problem is to make the unacceptable outcome impossible or detectable without making the acceptable path unusable. That means deterministic checks where the rule is knowable, classifiers only where it is not, and a record of every block so the policy can be argued with.
Mental model
Guardrails are the bollards at a station entrance, not the announcement asking people to walk carefully. The metaphor breaks because bollards are fixed geometry, while policy is contextual: a check that is correct today can be wrong after a rule change and nothing physical stops it.
How it works
Event · step 1 of 8
Input arrives
Entry: a support ticket body plus a retrieved article. The text may contain instructions; the harness treats all of it as data, never as policy.
In state
- Input
Next: Input check allows
- Policy checks
- 0
- Effects committed
- 0
- Blocks logged
- 0
Figure. Eight states. The input check and the argument check can stop a proposal before anything happens; the output check can only report what already happened. Blocked is a state the run reaches instead of the effect, not a message appended after it.
What to notice. Policy is data the harness evaluates, not a sentence the model is asked to obey. Deterministic checks cover amounts, recipients and data classes; a classifier is only for categories a rule cannot express. Redact and retry outcomes are omitted: both leave the run in a state with a modified payload rather than at Blocked, and escalation shares Blocked because it also stops before the effect. The trips, thresholds, allowlist and refund amounts are illustrative, and added latency and false-positive budgets are omitted.
Text version
A state machine with eight states. Input goes to Input check on content; the check goes to Propose on allow and to Blocked on deny. Propose goes to Arg check on a tool call; the argument check goes to Effect on allow and to Blocked on deny, the same terminal state an escalation uses. Effect goes to Delivered on a passing output scan, and to Too late on a guard placed after the effect. The trace walks Input, Input check, Propose, Arg check, Effect, Delivered, then the blocked 5,000 refund, then Too late. Counters show up to four policy checks, one effect committed and one block logged.
- Write the policy as concrete prohibitions and obligations: which data classes, which recipients, which amounts.
- Place a deterministic check where the effect occurs: validate tool arguments before the call and scan output before the response.
- Add a classifier only for categories a rule cannot express, and measure its precision on your own traffic.
- Decide per check whether to block, redact, retry or escalate, and default to deny for irreversible actions.
- Log the check id, the decision, the model version and a redacted excerpt for every trip.
- Replay a sample of blocked and allowed traffic monthly to find drift and false positives.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Prompt-level instruction | Cheap, no extra latency | Bypassed by injection or drift | Input is untrusted or the action is irreversible |
| Deterministic validator | Predictable, testable, fast | Only covers rules you can state | Policy is about meaning or intent |
| Classifier check | Covers fuzzy categories | Latency, cost and false positives | The category is rare in your traffic |
| Block on any trip | Strongest safety | Availability loss and user frustration | Checks are noisy or uncalibrated |
Good for
Cheap, no extra latency
Cost
Bypassed by injection or drift
Fails when
Input is untrusted or the action is irreversible
- Input is untrusted or the action is irreversible
- Policy is about meaning or intent
- The category is rare in your traffic
- Checks are noisy or uncalibrated
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Prompt-level instruction is good for Cheap, no extra latency, costs Bypassed by injection or drift, and fails when Input is untrusted or the action is irreversible. Deterministic validator is good for Predictable, testable, fast, costs Only covers rules you can state, and fails when Policy is about meaning or intent. Classifier check is good for Covers fuzzy categories, costs Latency, cost and false positives, and fails when The category is rare in your traffic. Block on any trip is good for Strongest safety, costs Availability loss and user frustration, and fails when Checks are noisy or uncalibrated
Failure modes
- Guardrail after the side effect. The email is sent and then the output is screened. Debug: trace the order of check and effect for each tool and assert the check precedes the commit.
- Uncalibrated classifier. A large share of legitimate requests are blocked and users route around the product. Debug: sample a hundred trips and label them by hand to get precision.
- Policy only in the prompt. An injected document tells the model to ignore the rule and it does. Debug: replay a known injection string and confirm the deterministic layer still blocks.
In production
Track block rate per check, false-positive rate from sampled reviews and added p95 latency. Version the policy alongside the prompts and alert when a check's trip rate moves sharply in either direction.
Common misconceptions
People think: A safety prompt is a guardrail.
Actually: A prompt shifts the distribution of outputs; it cannot make an action impossible, so it is a mitigation rather than a control.
People think: Guardrails are only for content moderation.
Actually: The most valuable guardrails constrain actions: amounts, recipients, scopes and data classes.
Interview angle
Strong answers start from the harm, place a control at the point of effect, then discuss calibration and the false-positive cost, distinguishing preventive checks from detective logging. Weak answers describe a moderation call at the end of the pipeline and call the system safe.
Sources
- OWASP, Top 10 for Large Language Model Applications — excessive agency and insecure output handling as named failure classes.
- Greshake et al., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — why retrieved content can override prompt-level instructions.
- NIST, AI Risk Management Framework — controls, measurement and governance vocabulary for risk.