Why it matters
One call has to do everything at once and gives no place to intervene. A chain lets the application inspect an intermediate result, branch on it, retry a failed step or call a tool between steps. That control is valuable when steps have different reliability or need different context. The engineering problem is that each seam is an untyped interface unless the application defines one, and prose is a poor carrier of state.
Mental model
A chain is a relay race: each runner hands over a baton at a marked point. The comparison breaks because the baton can be dropped silently, and the next runner will keep going with an empty hand. No official notices until the final result is wrong.
How it works
One call's prompt · illustrative tokens
1,600 input+ 300 reserve= 1,900 of 2,000
100 free
Step 1One call
System, task, document and question: 1,600 tokens and one place to fail. Everything the answer needs is present, and nothing can be inspected in between.
Output reserve
300One call: 1,600 input tokens, 100 free.
Pick a section to read its token count, its role and its state.
Figure. A chain re-sends a prompt at every step. The single call sends 1,600 input tokens; a three-step chain sends 2,750 — the document is paid for twice, and the extra 1,150 tokens buy two checkpoints, not a longer prompt.
What to notice. Counts are authored for a 2,000-token window with a 300-token answer reserve, across one call and a three-step chain; no model was called and no latency is measured. Each step draws one call's own prompt against the same window, so the shapes are comparable but not additive. Retries, parallel branches and the multi-turn resend of assistant text are omitted.
Text version
A 2,000-token window with a 300-token answer reserve, so 1,700 tokens are available. The default first step is one call: system 200, task 120, document 1,200 and question 80, for 1,600 input tokens and 100 free. The second step is call one of a chain, extracting fields: the same document is sent again with a 160-token extraction instruction, for 1,640 tokens and 60 free. The third step is call two, reasoning over the extracted state: system 200, reason instruction 120 and a 90-token state object, for 410 tokens and 1,290 free. The fourth step is call three, writing: system 200, format instruction 150, the state 90 and 260 tokens of findings, for 700 tokens. The fifth step replaces the state at call two with a 400-token prose hand-off: 720 tokens instead of 410. Selecting a section shows its count and role.
- Write the steps as a state machine with explicit inputs and outputs, not as a list of prompts.
- Define the state object passed between steps, with typed fields and a validation rule for each.
- Decide which steps can fail independently and where a retry or a branch belongs.
- Keep the model's prose out of the state unless it is the deliverable, and extract the fields you need.
- Set a total deadline and a per-step budget, and stop the chain when the deadline is exceeded.
- Compare the chain with a single-call baseline on quality, latency and cost before keeping it.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Two-step chain | Checkpoint and retry | Double latency | The task fits in one call |
| Long chain | Each step simple and observable | Compounding errors and orchestration cost | The steps are not truly independent |
| Structured state | Testable seams | Extraction code and schema maintenance | The intermediate result is inherently free-form |
| Prose hand-off | No extraction code | Silent information loss | A later step needs a specific field |
Good for
Checkpoint and retry
Cost
Double latency
Fails when
The task fits in one call
- The task fits in one call
- The steps are not truly independent
- The intermediate result is inherently free-form
- A later step needs a specific field
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Two-step chain is good for Checkpoint and retry, costs Double latency, and fails when The task fits in one call. Long chain is good for Each step simple and observable, costs Compounding errors and orchestration cost, and fails when The steps are not truly independent. Structured state is good for Testable seams, costs Extraction code and schema maintenance, and fails when The intermediate result is inherently free-form. Prose hand-off is good for No extraction code, costs Silent information loss, and fails when A later step needs a specific field
Failure modes
- Seam mismatch. Step two receives a summary where it expected a list and silently does nothing. Debug: validate the state schema between steps and log the payload at each seam.
- Retry amplification. A retried step re-runs downstream calls, multiplying cost. Debug: trace call counts per chain and compare with the number of user requests.
- Chain without a deadline. A slow step holds the whole request past the user's patience. Debug: record per-step duration and check which step consumes the budget.
In production
Trace chains as a single transaction with per-step spans, and alert on step-level failure rate and total duration. Cap steps and total tokens per chain.
Common misconceptions
People think: Chaining is inherently more accurate.
Actually: It adds checkpoints, not capability; errors can compound and hand-offs can lose information.
People think: A chain is an agent.
Actually: A chain has fixed control flow chosen by the developer; an agent chooses its own steps.
Interview angle
Explain what a checkpoint buys, then describe the state contract and the deadline that keeps the chain bounded. Strong answers compare against a single call and name the seam where errors appear. Weak answers chain prompts for their own sake and cannot say what the intermediate format guarantees.
Sources
- Anthropic, Building effective agents — prompt chaining as a workflow pattern and when to avoid it.
- Madaan et al., Self-Refine — iterative generate-critique-revise loops.
- Yao et al., ReAct — interleaving reasoning with actions across steps.