Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 10 of 26

Prompt chaining

0 of 26 complete

Why it matters

One call has to do everything at once and gives no place to intervene. A chain lets the application inspect an intermediate result, branch on it, retry a failed step or call a tool between steps. That control is valuable when steps have different reliability or need different context. The engineering problem is that each seam is an untyped interface unless the application defines one, and prose is a poor carrier of state.

Mental model

A chain is a relay race: each runner hands over a baton at a marked point. The comparison breaks because the baton can be dropped silently, and the next runner will keep going with an empty hand. No official notices until the final result is wrong.

How it works

What a chain costsInteractive

One call's prompt · illustrative tokens

1,600 input+ 300 reserve= 1,900 of 2,000

100 free

Step 1/5 · One call

Step 1One call

System, task, document and question: 1,600 tokens and one place to fail. Everything the answer needs is present, and nothing can be inspected in between.

reserve 3001,600 tokens

Output reserve

300

One call: 1,600 input tokens, 100 free.

output reservefixedBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. A chain re-sends a prompt at every step. The single call sends 1,600 input tokens; a three-step chain sends 2,750 — the document is paid for twice, and the extra 1,150 tokens buy two checkpoints, not a longer prompt.

What to notice. Counts are authored for a 2,000-token window with a 300-token answer reserve, across one call and a three-step chain; no model was called and no latency is measured. Each step draws one call's own prompt against the same window, so the shapes are comparable but not additive. Retries, parallel branches and the multi-turn resend of assistant text are omitted.

Text version

A 2,000-token window with a 300-token answer reserve, so 1,700 tokens are available. The default first step is one call: system 200, task 120, document 1,200 and question 80, for 1,600 input tokens and 100 free. The second step is call one of a chain, extracting fields: the same document is sent again with a 160-token extraction instruction, for 1,640 tokens and 60 free. The third step is call two, reasoning over the extracted state: system 200, reason instruction 120 and a 90-token state object, for 410 tokens and 1,290 free. The fourth step is call three, writing: system 200, format instruction 150, the state 90 and 260 tokens of findings, for 700 tokens. The fifth step replaces the state at call two with a 400-token prose hand-off: 720 tokens instead of 410. Selecting a section shows its count and role.

  1. Write the steps as a state machine with explicit inputs and outputs, not as a list of prompts.
  2. Define the state object passed between steps, with typed fields and a validation rule for each.
  3. Decide which steps can fail independently and where a retry or a branch belongs.
  4. Keep the model's prose out of the state unless it is the deliverable, and extract the fields you need.
  5. Set a total deadline and a per-step budget, and stop the chain when the deadline is exceeded.
  6. Compare the chain with a single-call baseline on quality, latency and cost before keeping it.

Trade-offs

ChoiceGood forCostFails when
Two-step chainCheckpoint and retryDouble latencyThe task fits in one call
Long chainEach step simple and observableCompounding errors and orchestration costThe steps are not truly independent
Structured stateTestable seamsExtraction code and schema maintenanceThe intermediate result is inherently free-form
Prose hand-offNo extraction codeSilent information lossA later step needs a specific field
Trade-off mapInteractive

Good for

Checkpoint and retry

Cost

Double latency

Fails when

The task fits in one call

  1. The task fits in one call
  2. The steps are not truly independent
  3. The intermediate result is inherently free-form
  4. A later step needs a specific field

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Two-step chain is good for Checkpoint and retry, costs Double latency, and fails when The task fits in one call. Long chain is good for Each step simple and observable, costs Compounding errors and orchestration cost, and fails when The steps are not truly independent. Structured state is good for Testable seams, costs Extraction code and schema maintenance, and fails when The intermediate result is inherently free-form. Prose hand-off is good for No extraction code, costs Silent information loss, and fails when A later step needs a specific field

Failure modes

  • Seam mismatch. Step two receives a summary where it expected a list and silently does nothing. Debug: validate the state schema between steps and log the payload at each seam.
  • Retry amplification. A retried step re-runs downstream calls, multiplying cost. Debug: trace call counts per chain and compare with the number of user requests.
  • Chain without a deadline. A slow step holds the whole request past the user's patience. Debug: record per-step duration and check which step consumes the budget.

In production

Trace chains as a single transaction with per-step spans, and alert on step-level failure rate and total duration. Cap steps and total tokens per chain.

Common misconceptions

People think: Chaining is inherently more accurate.

Actually: It adds checkpoints, not capability; errors can compound and hand-offs can lose information.

People think: A chain is an agent.

Actually: A chain has fixed control flow chosen by the developer; an agent chooses its own steps.

Interview angle

Explain what a checkpoint buys, then describe the state contract and the deadline that keeps the chain bounded. Strong answers compare against a single call and name the seam where errors appear. Weak answers chain prompts for their own sake and cannot say what the intermediate format guarantees.

Sources