Why it matters
Some tasks cannot be done in one step. Comparing two contract clauses, computing a refund across partial periods, or checking whether a set of constraints is satisfiable all require intermediate results. A model that must emit the answer immediately has to compress that computation into a single token decision.
Chain-of-thought changes the shape of the computation: the model writes intermediate values, and those values become context for later tokens. This is why it improves accuracy on multi-step tasks and why it does nothing for tasks that are already single-step. It is a technique with a scope, not a universal quality switch.
Mental model
Scratch paper during an exam. Writing the intermediate arithmetic frees working memory and makes errors visible, so the final answer is more likely to be right.
The analogy breaks because you can trust that the scratch paper reflects what the student did. A model's written reasoning is generated text like any other: it is produced by the same next-token process, and it may be a post-hoc rationalisation rather than the causal path to the answer. Treat the trace as evidence about the output, not as a window into the mechanism.
How it works
- 1Decide whether the task genuinely needs decomposition; if not, skip the technique.
- 2Either include worked examples that show the steps, or add a short instruction to reason before answering.
- 3Let the model produce the intermediate steps in a separate region from the final answer.
- 4Parse the final answer with a strict extractor, or require a structured field.
- 5Optionally sample several independent chains and aggregate the answers.
- 6Evaluate accuracy with and without the technique on a labelled set.
One request · illustrative tokens
260 input+ 600 reserve= 860 of 2,400
1,540 free
Step 1Direct answer
260 input tokens and a 600-token completion budget, of which a direct answer needs about 90. There is no trace and nothing to spend the reserve on.
Output reserve
600One call, one answer, 600 tokens of completion budget mostly unused.
Pick a section to read its token count, its role and its state.
Figure. The trace is output, and the reserve that holds it is subtracted from the prompt. Zero-shot costs 12 input tokens; few-shot costs 840; and reserving 1,400 tokens for a reasoning trace leaves 1,000 for a prompt that already needs 1,100.
What to notice. Input counts are authored for a 2,400-token window; no model was run, no benchmark is reported and no provider's reasoning budget is claimed. The trace length appears only as a reserve, because a generated trace is not prompt input. Self-consistency is drawn as a count under the strip: the window holds one sample, not five.
Text version
A 2,400-token window with a 600-token completion reserve, so 1,800 tokens are available for the prompt. The default first step is a direct answer: system 120 and question 140, for 260 input tokens and 1,540 free. The second step adds a 12-token think-step-by-step instruction: 272 tokens. The third step adds 840 tokens of worked reasoning chains as demonstrations: 1,100 tokens and 700 free. The fourth step raises the completion reserve to 1,400 tokens for a reasoning trace: the prompt space falls to 1,000 and the 1,100-token prompt runs 100 over, with the demonstration cells crossing the cap rule. The fifth step returns to a 600-token reserve with the demonstrations in place, and a note says five samples resend 1,100 tokens five times. Selecting a section shows its count and role.
Few-shot chain-of-thought was the original formulation: demonstrations include a question, a worked reasoning chain, and the answer. The model imitates the structure. Wei et al. showed gains on arithmetic, commonsense and symbolic reasoning benchmarks, with the effect emerging at larger model scales.
Zero-shot chain-of-thought is the cheaper variant: append an instruction such as "Let's think step by step" and let the model generate its own chain. Kojima et al. found this alone improved performance on reasoning benchmarks, which made the technique accessible without curated exemplars.
Self-consistency replaces a single chain with several sampled chains and takes the majority final answer. Wang et al. showed that aggregating diverse reasoning paths improves accuracy over greedy decoding on arithmetic and commonsense tasks. The cost is linear in the number of samples, and the method only works when the final answer can be compared exactly.
Reasoning models internalise the technique. They produce hidden reasoning tokens before the visible answer, and providers expose controls for reasoning effort rather than requiring you to prompt for steps. OpenAI's reasoning guide advises simpler prompts for these models; adding your own "think step by step" instructions is usually redundant and can conflict with the trained behaviour.
Quick facts
- Helps
- Multi-step reasoning
- Hurts
- Simple extraction, latency budgets
- Cost
- Output tokens, often the expensive kind
- Trace fidelity
- Not guaranteed
- Aggregation
- Self-consistency over samples
- Modern default
- Provider-managed reasoning
Self-consistency, in numbers
If a single chain is correct with probability and chains are independent, majority voting over samples helps when and the errors are not correlated. The cost is full generations:
In practice chains are correlated because they share the prompt, so the gain is smaller than the independence assumption suggests. Measure it; do not assume it.
A minimal self-consistency harness
from collections import Counter
def self_consistency(question: str, call_model, samples: int = 5) -> str:
answers = []
for _ in range(samples):
text = call_model(question, temperature=0.7)
answer = extract_final_answer(text) # your parser, not the model's prose
if answer is not None:
answers.append(answer)
if not answers:
raise ValueError("no parseable answer")
return Counter(answers).most_common(1)[0][0]Two details decide whether this works. The extractor must be strict, because a chain can mention several numbers before the final one. And sampling must actually produce diversity; temperature zero gives you the same chain repeatedly and the vote is meaningless.
Trade-offs
| Approach | Good for | Cost | Fails when |
|---|---|---|---|
| Direct answer | Extraction, classification, short factual replies | Lowest latency and tokens | The task needs intermediate computation |
| Zero-shot CoT | Quick improvement on reasoning tasks | More output tokens | The answer format must be terse or strict |
| Few-shot CoT | Tasks with a known solution pattern | Example tokens plus longer output | Examples encode a wrong or brittle method |
| Self-consistency | Verifiable answers with high variance | k full generations | Answers cannot be compared automatically |
| Reasoning model | Hard multi-step problems | Higher price and latency | The task is simple or the budget is tight |
Good for
Extraction, classification, short factual replies
Cost
Lowest latency and tokens
Fails when
The task needs intermediate computation
- The task needs intermediate computation
- The answer format must be terse or strict
- Examples encode a wrong or brittle method
- Answers cannot be compared automatically
- The task is simple or the budget is tight
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Direct answer is good for Extraction, classification, short factual replies, costs Lowest latency and tokens, and fails when The task needs intermediate computation. Zero-shot CoT is good for Quick improvement on reasoning tasks, costs More output tokens, and fails when The answer format must be terse or strict. Few-shot CoT is good for Tasks with a known solution pattern, costs Example tokens plus longer output, and fails when Examples encode a wrong or brittle method. Self-consistency is good for Verifiable answers with high variance, costs k full generations, and fails when Answers cannot be compared automatically. Reasoning model is good for Hard multi-step problems, costs Higher price and latency, and fails when The task is simple or the budget is tight
Failure modes
Unfaithful explanations. The chain sounds coherent but the answer was driven by a feature the chain never mentions. Turpin et al. demonstrated this with biasing cues: changing an irrelevant feature shifted the answer while the stated reasoning did not acknowledge it. Debug by testing whether the answer changes when you alter a cue that should not matter, and by validating the final answer independently.
Answer parsing errors. The model states an intermediate value and the extractor grabs it. Debug by requiring the final answer in a named field and by failing closed when the field is missing.
Cost and latency blow-up. Reasoning tokens are output tokens, and self-consistency multiplies them. Debug by measuring tokens per request before and after, and by capping the reasoning budget where the provider exposes one.
Sensitive content in traces. The chain may restate retrieved PII, internal identifiers or policy text. Debug by inspecting stored traces, not just user-visible answers, and redact before logging.
Symptom
The chain sounds coherent but the answer was driven by a feature the chain never mentions. Turpin et al. demonstrated this with biasing cues: changing an irrelevant feature shifted the answer while the stated reasoning did not acknowledge it.
Debug step
Testing whether the answer changes when you alter a cue that should not matter, and by validating the final answer independently.
Figure. Each failure with its symptom and the check that would confirm it.
What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.
Text version
Failure modes: Unfaithful explanations: The chain sounds coherent but the answer was driven by a feature the chain never mentions. Turpin et al. demonstrated this with biasing cues: changing an irrelevant feature shifted the answer while the stated reasoning did not acknowledge it. Debug by testing whether the answer changes when you alter a cue that should not matter, and by validating the final answer independently. | Answer parsing errors: The model states an intermediate value and the extractor grabs it. Debug by requiring the final answer in a named field and by failing closed when the field is missing. | Cost and latency blow-up: Reasoning tokens are output tokens, and self-consistency multiplies them. Debug by measuring tokens per request before and after, and by capping the reasoning budget where the provider exposes one. | Sensitive content in traces: The chain may restate retrieved PII, internal identifiers or policy text. Debug by inspecting stored traces, not just user-visible answers, and redact before logging.
In production
Separate the reasoning region from the answer region in your output contract, and store them separately. Cap reasoning length where the provider allows it. Do not show raw chains to end users by default; summarise the justification instead, and keep the raw trace for debugging under your data policy. Evaluate with and without the technique on the same set, because the benefit is task-specific and a model upgrade can remove it.
Common misconceptions
People think: Chain-of-thought reveals the model's actual computation.
Actually: It reveals generated text that often correlates with the answer. Research shows the stated reasoning can omit the factors that actually changed the output.
People think: Chain-of-thought improves every task.
Actually: It helps tasks that need decomposition. For extraction, classification and simple lookups it adds cost and latency with no accuracy gain, and can hurt format compliance.
People think: Reasoning models need explicit step-by-step instructions.
Actually: They are trained to reason internally. Provider guidance is to use simpler prompts and control effort, not to add chain-of-thought instructions.
Interview angle
Expect "the model gets multi-step maths wrong — how do you fix it". The strong answer decides whether the task needs decomposition, proposes a trace separated from the answer, checks faithfulness rather than trusting the chain, and measures cost. The weak answer says "add think step by step" and stops. See the prompting bank and the evaluation bank.
Sources
- Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — the original few-shot formulation.
- Kojima et al., Large Language Models are Zero-Shot Reasoners — zero-shot chain-of-thought.
- Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models — sampling and majority voting.
- Turpin et al., Language Models Don't Always Say What They Think — unfaithful chain-of-thought explanations.
Related
- Reasoning models — the trained alternative.
- Prompt decomposition — splitting the task instead of the reasoning.
- Self-consistency and sampling — the knobs that make voting work.
- LLM-as-judge — evaluating reasoning outputs.
- Prompt engineering — where this technique sits.