Why it matters
A multi-step agent is a distributed system: it calls slow services, waits for humans and can run for hours. Ordinary code loses its stack on restart, so recovery means starting over, which duplicates effects. Durable execution engines record the history of completed steps and replay the workflow to reconstruct state, so only unfinished work is retried.
Mental model
Durable execution is a journal replay: the workflow is the script, the history is what already happened, and recovery re-reads the script against the journal. The metaphor breaks because a replay assumes the script is unchanged, while a deploy changes it and the same journal must still produce a consistent state.
How it works
Event · step 1 of 8
Workflow schedules
Orchestration schedules charge_card with key order-8412. The workflow itself makes no network call; it only records the decision to run an activity.
In state
- Workflow
Next: Activity attempt 1
- Attempts
- 0
- Duplicate effects
- 0
Figure. The workflow schedules activities, each result is appended to history, and a transient timeout becomes one retry rather than a second charge. After the worker dies, replay reads history and skips the completed activity.
What to notice. Only orchestration is replayed; the activity's side effect is not. Determinism is what makes the journal match the script, and the idempotency key is what makes the retried call safe when the first attempt actually committed. Timeout and backoff values are illustrative.
Text version
A state machine with seven states. Workflow schedules Activity on schedule. Activity goes to History on recorded, and back from History on next. Activity goes to Retry on error, and Retry returns to Activity on attempt 2. History goes to Crash on dies. Crash goes to Resume on restart. Resume returns to Activity on skip and goes to Completed on replay. Workflow also goes to Completed on ends. The trace walks Workflow, Activity, Retry, Activity, History, Crash, Resume, Completed. Counters show attempts rising to two and duplicate side effects staying at zero.
- Split the workflow into deterministic orchestration and activities that perform side effects.
- Make each activity idempotent or keyed, because it may be retried after a partial failure.
- Record results in the history as each step completes, including timers and external signals.
- Configure retry policy per activity: attempts, backoff, and which errors are terminal.
- Version the workflow before changing its sequence, keeping old paths for in-flight runs.
- Test recovery by killing a worker mid-run and confirming the run completes without duplicate effects.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Durable workflow engine | Survives crashes and long waits | Engine operation and programming constraints | The team cannot adopt deterministic workflow code |
| Database-backed job table | Simple and familiar | Hand-rolled retries, leases and recovery | Steps are long or human waits are common |
| In-process with retries | Least infrastructure | Work lost on restart | Runs exceed a process lifetime |
| Synchronous chain | Easiest to reason about | No recovery, timeouts cascade | Any step is slow or unreliable |
Good for
Survives crashes and long waits
Cost
Engine operation and programming constraints
Fails when
The team cannot adopt deterministic workflow code
- The team cannot adopt deterministic workflow code
- Steps are long or human waits are common
- Runs exceed a process lifetime
- Any step is slow or unreliable
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Durable workflow engine is good for Survives crashes and long waits, costs Engine operation and programming constraints, and fails when The team cannot adopt deterministic workflow code. Database-backed job table is good for Simple and familiar, costs Hand-rolled retries, leases and recovery, and fails when Steps are long or human waits are common. In-process with retries is good for Least infrastructure, costs Work lost on restart, and fails when Runs exceed a process lifetime. Synchronous chain is good for Easiest to reason about, costs No recovery, timeouts cascade, and fails when Any step is slow or unreliable
Failure modes
- Non-deterministic workflow code. Replay takes a different branch and the history mismatches. Debug: replay a recorded history in a test and assert no non-determinism error.
- Duplicate activity. A retried activity charges a card twice. Debug: check the activity's idempotency key against the downstream record.
- Version collision. A deploy changes the step order and in-flight runs fail to replay. Debug: run a canary with an old history fixture before rolling out workflow changes.
In production
Monitor workflow failure rate, retries per activity and the age of the oldest running workflow. Rehearse a worker kill during a run and confirm recovery completes without duplicates.
Common misconceptions
People think: Durable execution means the agent never fails.
Actually: It means the run resumes; the failing step still fails until the underlying cause changes.
People think: Any code can be made durable by wrapping it.
Actually: Workflow code must be deterministic under replay, which rules out direct clock, randomness and network calls in orchestration.
Interview angle
Strong answers separate orchestration from activities, explain replay and why determinism is required, and raise versioning for in-flight runs. Weak answers describe retries and a job table without explaining how state is reconstructed after a crash.
Sources
- Temporal documentation — event history, deterministic workflow code and activity retries.
- AWS Step Functions Developer Guide — state-based orchestration with retries and error handling.