Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 21 of 28

Reasoning models

0 of 28 complete

Why it matters

Some tasks fail not because the model lacks knowledge but because it answers before working through the steps. A single forward pass per token gives no scratch space. Reasoning models are trained, often with reinforcement learning on verifiable problems, to use generated tokens as working memory. That buys accuracy on maths, code and planning, and introduces a new cost profile: long, variable, sometimes hidden generation.

Mental model

A reasoning model is a colleague who goes away to work the problem on paper before replying. The image breaks because you cannot always see the paper, and the written working may be a plausible reconstruction rather than the route actually taken.

How it works

Mechanism, step by stepInteractive
  1. 1Classify the task as multi-step and verifiable, or as a lookup, extraction or style task where extra reasoning adds cost without benefit.
  2. 2Send the problem with a clear success criterion and let the model manage its own reasoning rather than prescribing steps.
  3. 3Read the final answer only, and do not parse the reasoning trace as structured data.
  4. 4Budget for reasoning tokens in both cost and latency, since they are billed and generated before the answer.
  5. 5Set a timeout that tolerates long reasoning, and decide whether to retry or fall back to a faster model.
  6. 6Evaluate on a set with known answers so accuracy gains can be compared against the added cost.

Step 1 of 6

Classify the task as multi-step and verifiable, or as a lookup, extraction or style task where extra reasoning adds cost without benefit.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Classify the task as multi-step and verifiable, or as a lookup, extraction or style task … 2. Send the problem with a clear success criterion and let the model manage its own reasonin… 3. Read the final answer only, and do not parse the reasoning trace as structured data. 4. Budget for reasoning tokens in both cost and latency, since they are billed and generated… 5. Set a timeout that tolerates long reasoning, and decide whether to retry or fall back to … 6. Evaluate on a set with known answers so accuracy gains can be compared against the added …

  1. Classify the task as multi-step and verifiable, or as a lookup, extraction or style task where extra reasoning adds cost without benefit.
  2. Send the problem with a clear success criterion and let the model manage its own reasoning rather than prescribing steps.
  3. Read the final answer only, and do not parse the reasoning trace as structured data.
  4. Budget for reasoning tokens in both cost and latency, since they are billed and generated before the answer.
  5. Set a timeout that tolerates long reasoning, and decide whether to retry or fall back to a faster model.
  6. Evaluate on a set with known answers so accuracy gains can be compared against the added cost.

Trade-offs

ChoiceGood forCostFails when
Reasoning model on hard tasksBetter multi-step accuracyHigh and variable latencyThe path is interactive and the budget is tight
Standard model with explicit stepsCheaper, predictableWeaker on problems needing searchThe problem requires backtracking
Reasoning model everywhereOne integration, simple routingPays reasoning cost on trivial requestsTraffic is dominated by simple queries
Cascade from fast to reasoningCost control with a quality floorEscalation logic and two evaluationsYou cannot tell when the fast answer is wrong
Trade-off mapInteractive

Good for

Better multi-step accuracy

Cost

High and variable latency

Fails when

The path is interactive and the budget is tight

  1. The path is interactive and the budget is tight
  2. The problem requires backtracking
  3. Traffic is dominated by simple queries
  4. You cannot tell when the fast answer is wrong

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Reasoning model on hard tasks is good for Better multi-step accuracy, costs High and variable latency, and fails when The path is interactive and the budget is tight. Standard model with explicit steps is good for Cheaper, predictable, costs Weaker on problems needing search, and fails when The problem requires backtracking. Reasoning model everywhere is good for One integration, simple routing, costs Pays reasoning cost on trivial requests, and fails when Traffic is dominated by simple queries. Cascade from fast to reasoning is good for Cost control with a quality floor, costs Escalation logic and two evaluations, and fails when You cannot tell when the fast answer is wrong

Failure modes

  • Reasoning tax on simple work. Latency triples for extraction tasks that did not need reasoning. Debug: compare per-route latency and cost before and after routing the task to a reasoning model.
  • Trace treated as explanation. A summary of the reasoning is shown to users as justification and does not match the answer. Debug: check consistency between trace and answer on a sample, and do not present the trace as an audit record.
  • Timeout mid-reasoning. The request is cut off before an answer and retried, doubling cost. Debug: log completion status and reasoning token counts, and set timeouts from observed p99.

In production

Monitor reasoning token share, p95 latency and timeout rate per route. Cap spend per request and keep a non-reasoning fallback for when latency budgets are breached.

Common misconceptions

People think: The visible reasoning is what the model actually computed.

Actually: The trace is generated text and can be incomplete or unfaithful; only the answer and external checks are evidence.

People think: Reasoning models are strictly better.

Actually: They cost more and are slower, and on simple extraction or formatting tasks they can be no better or worse.

Interview angle

Explain test-time compute as a trade rather than a free upgrade, and separate the answer from the trace. Strong answers route tasks and budget latency explicitly. Weak answers claim the model thinks, present the trace as an explanation, and have no plan for the cost of long reasoning.

Sources