Skip to content
AI Engineer Guide.

concept · concepts · beginner · concept 8 of 12

Rules vs model decisions

0 of 12 complete

Why it matters

A model is the wrong tool for a decision with a known rule: it is slower, costs money and can be wrong in ways that are hard to reproduce. A rule is the wrong tool for a decision that depends on phrasing, context or perception. Most production systems need both, and the design question is the boundary between decisions that must be reproducible and those that can tolerate sampling.

Mental model

Rules are arithmetic; model decisions are a judgement call from an experienced colleague. The metaphor breaks because a colleague can explain the judgement and be corrected, while a model's rationale is generated after the decision and is not a reliable account of it.

How it works

Where the rule ends and the model beginsInteractive
highCost of a wrong decisionlow
structuredlanguage or perception
Ambiguity of the input →
VerdictChoose· recommended hereConditional· with the named cost

Step 1 of 4

The knowable corner

If you can write the rule and the inputs are structured, write the rule. A model here adds cost and variance to a decision that never needed judgement.

Ambiguity of the input: low · Cost of a wrong decision: low

Choose · Write the rule

Code is cheaper, faster and exact; keep the threshold out of the prompt.

Implement it with tests and keep it out of the prompt. A threshold stated in natural language is applied inconsistently.

Axes

Ambiguity of the input
structured → language or perception
Cost of a wrong decision
cheap to reverse → regulated or must be explained
Step 1/4 · The knowable corner

Figure. Ambiguity decides whether a model is needed; the cost of an error decides whether its output is a proposal or a commit. The top-right cell is where the model and the code both have a job.

What to notice. The axes are qualitative. No confidence threshold appears because the page does not name one; the fallback is a design choice for your system. The figure maps one decision at a time — a pipeline usually contains several cells — and it does not claim that a model's rationale explains its decision.

Text version

A two-by-two decision map. The horizontal axis is ambiguity of the input, from structured on the left to language or perception on the right. The vertical axis is the cost of a wrong decision, from cheap to reverse at the bottom to regulated or must be explained at the top. Bottom left, a structured input with a writable rule: write the rule and keep the threshold out of the prompt. Bottom right, ambiguous input with a recoverable mistake: let the model propose, constrain the output to a schema, and log which path each decision took. Top left, a structured input that must be explained: use a rule with a validator, because tests are the evidence a review will ask for. Top right, ambiguity with state-changing consequences: the model proposes and code disposes, with a validator before the commit and a defined low-confidence fallback. Four steps walk the cells in that order.

  1. List every decision in the flow and mark whether its inputs are structured and its rule is knowable.
  2. Implement the knowable ones as code with tests, and keep them out of the prompt.
  3. Route ambiguous decisions to the model and constrain its output to a schema.
  4. Validate model output with deterministic checks before it changes state.
  5. Define a fallback for low confidence: default, escalate, or ask for clarification.
  6. Log which path each decision took so rule and model behaviour can be compared.

Trade-offs

ChoiceGood forCostFails when
Pure rulesReproducible and auditableBrittle to inputs outside the ruleLanguage or perception is involved
Pure modelHandles varietyNon-reproducible and costlierThe decision is regulated or must be explained
Model proposes, code disposesClear audit point, both strengthsTwo layers to maintainThe latency budget is very tight
Rules with model fallbackCheap common pathFallback is under-testedThe long tail is most of the traffic
Trade-off mapInteractive

Good for

Reproducible and auditable

Cost

Brittle to inputs outside the rule

Fails when

Language or perception is involved

  1. Language or perception is involved
  2. The decision is regulated or must be explained
  3. The latency budget is very tight
  4. The long tail is most of the traffic

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Pure rules is good for Reproducible and auditable, costs Brittle to inputs outside the rule, and fails when Language or perception is involved. Pure model is good for Handles variety, costs Non-reproducible and costlier, and fails when The decision is regulated or must be explained. Model proposes, code disposes is good for Clear audit point, both strengths, costs Two layers to maintain, and fails when The latency budget is very tight. Rules with model fallback is good for Cheap common path, costs Fallback is under-tested, and fails when The long tail is most of the traffic

Failure modes

  • Rule in a prompt. A threshold stated in natural language is applied inconsistently. Debug: search prompts for numbers and move each into code with a test.
  • Model on a knowable rule. Cost and variance appear on a decision that never needed judgement. Debug: replay a sample through a rule and compare agreement.
  • Missing fallback. Low-confidence outputs are committed as if certain. Debug: histogram confidence and check what happens below the threshold.

In production

Monitor the share of decisions taken by each path and their disagreement rate. Review the fallback path with the same care as the main one, since it runs when things are already wrong.

Common misconceptions

People think: Models should decide everything once they are good enough.

Actually: A model is not a reason to discard a rule that is cheaper, faster and exact.

People think: Rules mean the system is not really AI.

Actually: Most AI products are mostly rules, with a model at the point where judgement is unavoidable.

Interview angle

Strong answers walk the flow and assign each decision to code or model by the knowability of the rule, then describe the validation and fallback. Weak answers treat it as a binary preference for ML over heuristics.

Sources