Why it matters
Some tasks fail not because the model lacks knowledge but because it answers before working through the steps. A single forward pass per token gives no scratch space. Reasoning models are trained, often with reinforcement learning on verifiable problems, to use generated tokens as working memory. That buys accuracy on maths, code and planning, and introduces a new cost profile: long, variable, sometimes hidden generation.
Mental model
A reasoning model is a colleague who goes away to work the problem on paper before replying. The image breaks because you cannot always see the paper, and the written working may be a plausible reconstruction rather than the route actually taken.
How it works
- 1Classify the task as multi-step and verifiable, or as a lookup, extraction or style task where extra reasoning adds cost without benefit.
- 2Send the problem with a clear success criterion and let the model manage its own reasoning rather than prescribing steps.
- 3Read the final answer only, and do not parse the reasoning trace as structured data.
- 4Budget for reasoning tokens in both cost and latency, since they are billed and generated before the answer.
- 5Set a timeout that tolerates long reasoning, and decide whether to retry or fall back to a faster model.
- 6Evaluate on a set with known answers so accuracy gains can be compared against the added cost.
Step 1 of 6
Classify the task as multi-step and verifiable, or as a lookup, extraction or style task where extra reasoning adds cost without benefit.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Classify the task as multi-step and verifiable, or as a lookup, extraction or style task … 2. Send the problem with a clear success criterion and let the model manage its own reasonin… 3. Read the final answer only, and do not parse the reasoning trace as structured data. 4. Budget for reasoning tokens in both cost and latency, since they are billed and generated… 5. Set a timeout that tolerates long reasoning, and decide whether to retry or fall back to … 6. Evaluate on a set with known answers so accuracy gains can be compared against the added …
- Classify the task as multi-step and verifiable, or as a lookup, extraction or style task where extra reasoning adds cost without benefit.
- Send the problem with a clear success criterion and let the model manage its own reasoning rather than prescribing steps.
- Read the final answer only, and do not parse the reasoning trace as structured data.
- Budget for reasoning tokens in both cost and latency, since they are billed and generated before the answer.
- Set a timeout that tolerates long reasoning, and decide whether to retry or fall back to a faster model.
- Evaluate on a set with known answers so accuracy gains can be compared against the added cost.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Reasoning model on hard tasks | Better multi-step accuracy | High and variable latency | The path is interactive and the budget is tight |
| Standard model with explicit steps | Cheaper, predictable | Weaker on problems needing search | The problem requires backtracking |
| Reasoning model everywhere | One integration, simple routing | Pays reasoning cost on trivial requests | Traffic is dominated by simple queries |
| Cascade from fast to reasoning | Cost control with a quality floor | Escalation logic and two evaluations | You cannot tell when the fast answer is wrong |
Good for
Better multi-step accuracy
Cost
High and variable latency
Fails when
The path is interactive and the budget is tight
- The path is interactive and the budget is tight
- The problem requires backtracking
- Traffic is dominated by simple queries
- You cannot tell when the fast answer is wrong
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Reasoning model on hard tasks is good for Better multi-step accuracy, costs High and variable latency, and fails when The path is interactive and the budget is tight. Standard model with explicit steps is good for Cheaper, predictable, costs Weaker on problems needing search, and fails when The problem requires backtracking. Reasoning model everywhere is good for One integration, simple routing, costs Pays reasoning cost on trivial requests, and fails when Traffic is dominated by simple queries. Cascade from fast to reasoning is good for Cost control with a quality floor, costs Escalation logic and two evaluations, and fails when You cannot tell when the fast answer is wrong
Failure modes
- Reasoning tax on simple work. Latency triples for extraction tasks that did not need reasoning. Debug: compare per-route latency and cost before and after routing the task to a reasoning model.
- Trace treated as explanation. A summary of the reasoning is shown to users as justification and does not match the answer. Debug: check consistency between trace and answer on a sample, and do not present the trace as an audit record.
- Timeout mid-reasoning. The request is cut off before an answer and retried, doubling cost. Debug: log completion status and reasoning token counts, and set timeouts from observed p99.
In production
Monitor reasoning token share, p95 latency and timeout rate per route. Cap spend per request and keep a non-reasoning fallback for when latency budgets are breached.
Common misconceptions
People think: The visible reasoning is what the model actually computed.
Actually: The trace is generated text and can be incomplete or unfaithful; only the answer and external checks are evidence.
People think: Reasoning models are strictly better.
Actually: They cost more and are slower, and on simple extraction or formatting tasks they can be no better or worse.
Interview angle
Explain test-time compute as a trade rather than a free upgrade, and separate the answer from the trace. Strong answers route tasks and budget latency explicitly. Weak answers claim the model thinks, present the trace as an explanation, and have no plan for the cost of long reasoning.
Sources
- OpenAI, o1 System Card — behaviour and safety evaluation of a reasoning model.
- DeepSeek-AI, DeepSeek-R1 — reinforcement learning that elicits long reasoning.
- Snell et al., Scaling LLM Test-Time Compute Optimally — when extra inference compute helps and when it does not.