Why it matters
A prompt that extracts, reasons and formats in one pass fails in ways that are hard to diagnose: a bad extraction and a bad inference look the same in the output. Decomposition gives each stage a contract, so a failure can be traced to the stage that produced it. The cost is real: more calls, more latency, and intermediate representations that may drop the detail a later stage needs.
Mental model
Decomposition is an assembly line: each station does one operation and passes a part on. The metaphor breaks because a misread part is not visibly defective, and later stations will confidently build on it. The defect surfaces only in the finished product, far from its cause.
How it works
Step 1 of 7
One prompt fails unattributably
The first attempt is a single call: extract, total, format. It is exact on 13 of 25 invoices, and the failures do not say which of the three jobs went wrong.
branch rejected · current
single-prompt baseline
One call, 1.7s p50, correct on 13 of 25 invoices. The failures are unattributable: a bad span and a bad total look the same in the JSON.
Under Task · invoice to 3 fields, total, JSON
Figure. A task split at its context and format boundaries: extraction, arithmetic and rendering, each with its own prompt and a gate on the hand-off. The single-prompt baseline it replaces is drawn as a sibling branch.
What to notice. Decomposition is an interface decision: every edge is a contract and every gate is a place a failure stops. The same edges carry the risk — a lossy hand-off and compounding error — which is why the pipeline is measured against the single prompt. Accuracies, latency and stage versions are illustrative; 0.95³ ≈ 0.86 is arithmetic on the stated per-stage rate, not a measurement.
Text version
A prompt pipeline drawn as a hierarchy. The root is an invoice task: extract three fields, compute a total and render JSON. Under it are three stages, each tagged with a 95 percent exact rate. Stage 1 extracts spans, with prompt extract.v3 at temperature zero and a gate that rejects any span not found verbatim in the text. Stage 2 computes the total, with prompt compute.v2 and a gate that requires the sum to be within one cent of the printed lines; a rejected child shows a summarised hand-off dropping the line amounts. Stage 3 renders JSON against schema v4 with a citation gate. Two further branches compare the single-prompt baseline, rejected at 52 percent exact match, with the three-stage result at 84 percent and three calls. A final step walks the comparison.
- Write the end-to-end task and mark the points where the required context or output format changes.
- Split only at those points, and define the input and output contract for each stage.
- Choose a representation that preserves what later stages need, such as verbatim spans rather than a summary.
- Test each stage against its own labelled examples before testing the pipeline.
- Add a validation gate between stages so a malformed intermediate result stops the chain.
- Measure the end-to-end score against a single-prompt baseline, including latency and cost.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Single prompt | One call, low latency | Failures are unattributable | The task mixes extraction, reasoning and formatting |
| Two or three stages | Localised failures, checkable steps | Hand-off loss and extra calls | The intermediate format cannot carry the detail |
| Many small stages | Each stage simple and testable | Compounding error and orchestration cost | The task is short and mostly linear |
| Deterministic code between stages | Removes model error at that step | Less flexible parsing | The transformation needs judgement |
Good for
One call, low latency
Cost
Failures are unattributable
Fails when
The task mixes extraction, reasoning and formatting
- The task mixes extraction, reasoning and formatting
- The intermediate format cannot carry the detail
- The task is short and mostly linear
- The transformation needs judgement
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Single prompt is good for One call, low latency, costs Failures are unattributable, and fails when The task mixes extraction, reasoning and formatting. Two or three stages is good for Localised failures, checkable steps, costs Hand-off loss and extra calls, and fails when The intermediate format cannot carry the detail. Many small stages is good for Each stage simple and testable, costs Compounding error and orchestration cost, and fails when The task is short and mostly linear. Deterministic code between stages is good for Removes model error at that step, costs Less flexible parsing, and fails when The transformation needs judgement
Failure modes
- Hand-off loss. A summary drops the identifier the next stage needs. Debug: diff the intermediate output against the source spans and check the required fields are present.
- Error compounding. Each stage is 95 percent accurate and the pipeline is far worse. Debug: measure per-stage accuracy and multiply, then compare with the observed end-to-end score.
- Stage contract drift. A prompt change alters the intermediate format and breaks the next stage. Debug: validate the intermediate against a schema and fail the request, not the parser.
In production
Log per-stage inputs and outputs with a shared request id, and alert on per-stage validation failures rather than only end-to-end errors. Cap the number of stages so latency stays within budget.
Common misconceptions
People think: Decomposition always improves accuracy.
Actually: It helps when sub-tasks conflict, and hurts when hand-offs lose information or errors compound.
People think: More stages mean more control.
Actually: Each stage adds a contract to maintain and a place for silent corruption.
Interview angle
Start from the failure you are trying to isolate, then split only at context or format boundaries and define each contract. Strong answers mention per-stage evaluation and error compounding. Weak answers split everything into steps and never compare the pipeline with a single-prompt baseline.
Sources
- Zhou et al., Least-to-Most Prompting — solving a task as a sequence of easier sub-problems.
- Wang et al., Plan-and-Solve Prompting — planning before execution as a decomposition strategy.
- Anthropic, Building effective agents — prompt chaining and decomposition patterns with their costs.