midReportedcurated
Your chain-of-thought prompt is not improving accuracy on a reasoning task. What do you check and what do you change?
30-second answer
- First check whether the task needs decomposition: chain-of-thought adds output tokens and latency and does little for lookup, classification or extraction.
- If the model is reasoning-trained, it already produces hidden reasoning, so an explicit think step by step instruction is redundant and can conflict with its trained behaviour.
- For a genuinely multi-step task, inspect traces for the first wrong step, then try self-consistency over several samples, decomposition, a worked example, or a calculator for arithmetic.
- Self-consistency only helps when single-chain accuracy is above half and errors are not correlated, and it costs one full generation per sample.
- Measure on a sliced eval set before and after, because rationales can be plausible post-hoc text that does not reflect the computation.
Likely follow-ups
Open a question to rehearse the answer.
How would you tell whether the written reasoning is faithful?
- Change an irrelevant cue that should not affect the answer and see whether the answer moves while the stated reasoning stays the same.
- Validate the final answer independently with a checker or a second method, because a coherent chain is generated text rather than an audit log.
- Compare the numbers in the chain against the final answer, since an extractor can also grab an intermediate value.
When is a reasoning model better value than a CoT prompt?
- It wins on hard multi-step problems where you would otherwise sample several chains, because the reasoning budget replaces prompt engineering and voting.
- It loses on simple extraction and classification, where the extra tokens add cost and latency without accuracy.
- Price the provider's reasoning tokens against k sampled chains, and keep CoT when you need visible traces for debugging.
Red flags
- Keeps adding phrases like think harder
- Judges the change from a handful of examples
Concepts this tests
Tags
Source: github.com
More in Prompting & context engineering
- You stuff 60 retrieved documents into one prompt and the model uses facts from the beginning and end but ignores a crucial fact in the middle. Diagnose it and fix it.
- An agent reads retrieved documents and tool outputs and can take actions. How do you defend against indirect prompt injection, and why can't a better system prompt solve it?
- Your JSON parsing works in development but fails intermittently in production even at temperature 0. Diagnose it and describe the durable fix.