Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 6 of 26

Chain-of-thought

0 of 26 complete

Why it matters

Some tasks cannot be done in one step. Comparing two contract clauses, computing a refund across partial periods, or checking whether a set of constraints is satisfiable all require intermediate results. A model that must emit the answer immediately has to compress that computation into a single token decision.

Chain-of-thought changes the shape of the computation: the model writes intermediate values, and those values become context for later tokens. This is why it improves accuracy on multi-step tasks and why it does nothing for tasks that are already single-step. It is a technique with a scope, not a universal quality switch.

Mental model

Scratch paper during an exam. Writing the intermediate arithmetic frees working memory and makes errors visible, so the final answer is more likely to be right.

The analogy breaks because you can trust that the scratch paper reflects what the student did. A model's written reasoning is generated text like any other: it is produced by the same next-token process, and it may be a post-hoc rationalisation rather than the causal path to the answer. Treat the trace as evidence about the output, not as a window into the mechanism.

How it works

  1. 1Decide whether the task genuinely needs decomposition; if not, skip the technique.
  2. 2Either include worked examples that show the steps, or add a short instruction to reason before answering.
  3. 3Let the model produce the intermediate steps in a separate region from the final answer.
  4. 4Parse the final answer with a strict extractor, or require a structured field.
  5. 5Optionally sample several independent chains and aggregate the answers.
  6. 6Evaluate accuracy with and without the technique on a labelled set.
Reasoning is paid from the same windowInteractive

One request · illustrative tokens

260 input+ 600 reserve= 860 of 2,400

1,540 free

Step 1/5 · Direct answer

Step 1Direct answer

260 input tokens and a 600-token completion budget, of which a direct answer needs about 90. There is no trace and nothing to spend the reserve on.

reserve 600260 tokens

Output reserve

600

One call, one answer, 600 tokens of completion budget mostly unused.

output reservefixedBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. The trace is output, and the reserve that holds it is subtracted from the prompt. Zero-shot costs 12 input tokens; few-shot costs 840; and reserving 1,400 tokens for a reasoning trace leaves 1,000 for a prompt that already needs 1,100.

What to notice. Input counts are authored for a 2,400-token window; no model was run, no benchmark is reported and no provider's reasoning budget is claimed. The trace length appears only as a reserve, because a generated trace is not prompt input. Self-consistency is drawn as a count under the strip: the window holds one sample, not five.

Text version

A 2,400-token window with a 600-token completion reserve, so 1,800 tokens are available for the prompt. The default first step is a direct answer: system 120 and question 140, for 260 input tokens and 1,540 free. The second step adds a 12-token think-step-by-step instruction: 272 tokens. The third step adds 840 tokens of worked reasoning chains as demonstrations: 1,100 tokens and 700 free. The fourth step raises the completion reserve to 1,400 tokens for a reasoning trace: the prompt space falls to 1,000 and the 1,100-token prompt runs 100 over, with the demonstration cells crossing the cap rule. The fifth step returns to a 600-token reserve with the demonstrations in place, and a note says five samples resend 1,100 tokens five times. Selecting a section shows its count and role.

Few-shot chain-of-thought was the original formulation: demonstrations include a question, a worked reasoning chain, and the answer. The model imitates the structure. Wei et al. showed gains on arithmetic, commonsense and symbolic reasoning benchmarks, with the effect emerging at larger model scales.

Zero-shot chain-of-thought is the cheaper variant: append an instruction such as "Let's think step by step" and let the model generate its own chain. Kojima et al. found this alone improved performance on reasoning benchmarks, which made the technique accessible without curated exemplars.

Self-consistency replaces a single chain with several sampled chains and takes the majority final answer. Wang et al. showed that aggregating diverse reasoning paths improves accuracy over greedy decoding on arithmetic and commonsense tasks. The cost is linear in the number of samples, and the method only works when the final answer can be compared exactly.

Reasoning models internalise the technique. They produce hidden reasoning tokens before the visible answer, and providers expose controls for reasoning effort rather than requiring you to prompt for steps. OpenAI's reasoning guide advises simpler prompts for these models; adding your own "think step by step" instructions is usually redundant and can conflict with the trained behaviour.

Quick facts

Helps
Multi-step reasoning
Hurts
Simple extraction, latency budgets
Cost
Output tokens, often the expensive kind
Trace fidelity
Not guaranteed
Aggregation
Self-consistency over samples
Modern default
Provider-managed reasoning

Self-consistency, in numbers

If a single chain is correct with probability pp and chains are independent, majority voting over kk samples helps when p>0.5p > 0.5 and the errors are not correlated. The cost is kk full generations:

costSC≈k⋅(Tin⋅pin+Tout,chain⋅pout)\text{cost}_{\text{SC}} \approx k \cdot \left(T_{\text{in}} \cdot p_{\text{in}} + T_{\text{out,chain}} \cdot p_{\text{out}}\right)

In practice chains are correlated because they share the prompt, so the gain is smaller than the independence assumption suggests. Measure it; do not assume it.

A minimal self-consistency harness

from collections import Counter
 
def self_consistency(question: str, call_model, samples: int = 5) -> str:
    answers = []
    for _ in range(samples):
        text = call_model(question, temperature=0.7)
        answer = extract_final_answer(text)   # your parser, not the model's prose
        if answer is not None:
            answers.append(answer)
 
    if not answers:
        raise ValueError("no parseable answer")
    return Counter(answers).most_common(1)[0][0]

Two details decide whether this works. The extractor must be strict, because a chain can mention several numbers before the final one. And sampling must actually produce diversity; temperature zero gives you the same chain repeatedly and the vote is meaningless.

Trade-offs

ApproachGood forCostFails when
Direct answerExtraction, classification, short factual repliesLowest latency and tokensThe task needs intermediate computation
Zero-shot CoTQuick improvement on reasoning tasksMore output tokensThe answer format must be terse or strict
Few-shot CoTTasks with a known solution patternExample tokens plus longer outputExamples encode a wrong or brittle method
Self-consistencyVerifiable answers with high variancek full generationsAnswers cannot be compared automatically
Reasoning modelHard multi-step problemsHigher price and latencyThe task is simple or the budget is tight
Trade-off mapInteractive

Good for

Extraction, classification, short factual replies

Cost

Lowest latency and tokens

Fails when

The task needs intermediate computation

  1. The task needs intermediate computation
  2. The answer format must be terse or strict
  3. Examples encode a wrong or brittle method
  4. Answers cannot be compared automatically
  5. The task is simple or the budget is tight

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Direct answer is good for Extraction, classification, short factual replies, costs Lowest latency and tokens, and fails when The task needs intermediate computation. Zero-shot CoT is good for Quick improvement on reasoning tasks, costs More output tokens, and fails when The answer format must be terse or strict. Few-shot CoT is good for Tasks with a known solution pattern, costs Example tokens plus longer output, and fails when Examples encode a wrong or brittle method. Self-consistency is good for Verifiable answers with high variance, costs k full generations, and fails when Answers cannot be compared automatically. Reasoning model is good for Hard multi-step problems, costs Higher price and latency, and fails when The task is simple or the budget is tight

Failure modes

Unfaithful explanations. The chain sounds coherent but the answer was driven by a feature the chain never mentions. Turpin et al. demonstrated this with biasing cues: changing an irrelevant feature shifted the answer while the stated reasoning did not acknowledge it. Debug by testing whether the answer changes when you alter a cue that should not matter, and by validating the final answer independently.

Answer parsing errors. The model states an intermediate value and the extractor grabs it. Debug by requiring the final answer in a named field and by failing closed when the field is missing.

Cost and latency blow-up. Reasoning tokens are output tokens, and self-consistency multiplies them. Debug by measuring tokens per request before and after, and by capping the reasoning budget where the provider exposes one.

Sensitive content in traces. The chain may restate retrieved PII, internal identifiers or policy text. Debug by inspecting stored traces, not just user-visible answers, and redact before logging.

Failure modes, one at a timeInteractive

Symptom

The chain sounds coherent but the answer was driven by a feature the chain never mentions. Turpin et al. demonstrated this with biasing cues: changing an irrelevant feature shifted the answer while the stated reasoning did not acknowledge it.

Debug step

Testing whether the answer changes when you alter a cue that should not matter, and by validating the final answer independently.

Figure. Each failure with its symptom and the check that would confirm it.

What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.

Text version

Failure modes: Unfaithful explanations: The chain sounds coherent but the answer was driven by a feature the chain never mentions. Turpin et al. demonstrated this with biasing cues: changing an irrelevant feature shifted the answer while the stated reasoning did not acknowledge it. Debug by testing whether the answer changes when you alter a cue that should not matter, and by validating the final answer independently. | Answer parsing errors: The model states an intermediate value and the extractor grabs it. Debug by requiring the final answer in a named field and by failing closed when the field is missing. | Cost and latency blow-up: Reasoning tokens are output tokens, and self-consistency multiplies them. Debug by measuring tokens per request before and after, and by capping the reasoning budget where the provider exposes one. | Sensitive content in traces: The chain may restate retrieved PII, internal identifiers or policy text. Debug by inspecting stored traces, not just user-visible answers, and redact before logging.

In production

Separate the reasoning region from the answer region in your output contract, and store them separately. Cap reasoning length where the provider allows it. Do not show raw chains to end users by default; summarise the justification instead, and keep the raw trace for debugging under your data policy. Evaluate with and without the technique on the same set, because the benefit is task-specific and a model upgrade can remove it.

Common misconceptions

People think: Chain-of-thought reveals the model's actual computation.

Actually: It reveals generated text that often correlates with the answer. Research shows the stated reasoning can omit the factors that actually changed the output.

People think: Chain-of-thought improves every task.

Actually: It helps tasks that need decomposition. For extraction, classification and simple lookups it adds cost and latency with no accuracy gain, and can hurt format compliance.

People think: Reasoning models need explicit step-by-step instructions.

Actually: They are trained to reason internally. Provider guidance is to use simpler prompts and control effort, not to add chain-of-thought instructions.

Interview angle

Expect "the model gets multi-step maths wrong — how do you fix it". The strong answer decides whether the task needs decomposition, proposes a trace separated from the answer, checks faithfulness rather than trusting the chain, and measures cost. The weak answer says "add think step by step" and stops. See the prompting bank and the evaluation bank.

Sources