Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 28 of 37

Durable execution

0 of 37 complete

Why it matters

A multi-step agent is a distributed system: it calls slow services, waits for humans and can run for hours. Ordinary code loses its stack on restart, so recovery means starting over, which duplicates effects. Durable execution engines record the history of completed steps and replay the workflow to reconstruct state, so only unfinished work is retried.

Mental model

Durable execution is a journal replay: the workflow is the script, the history is what already happened, and recovery re-reads the script against the journal. The metaphor breaks because a replay assumes the script is unchanged, while a deploy changes it and the same journal must still produce a consistent state.

How it works

Activity retried, history recorded, crash replayedInteractive
transitionfailurerepeat

Event · step 1 of 8

Workflow schedules

Orchestration schedules charge_card with key order-8412. The workflow itself makes no network call; it only records the decision to run an activity.

In state

  • Workflow

Next: Activity attempt 1

Attempts
0
Duplicate effects
0
Step 1/8 · Workflow schedules

Figure. The workflow schedules activities, each result is appended to history, and a transient timeout becomes one retry rather than a second charge. After the worker dies, replay reads history and skips the completed activity.

What to notice. Only orchestration is replayed; the activity's side effect is not. Determinism is what makes the journal match the script, and the idempotency key is what makes the retried call safe when the first attempt actually committed. Timeout and backoff values are illustrative.

Text version

A state machine with seven states. Workflow schedules Activity on schedule. Activity goes to History on recorded, and back from History on next. Activity goes to Retry on error, and Retry returns to Activity on attempt 2. History goes to Crash on dies. Crash goes to Resume on restart. Resume returns to Activity on skip and goes to Completed on replay. Workflow also goes to Completed on ends. The trace walks Workflow, Activity, Retry, Activity, History, Crash, Resume, Completed. Counters show attempts rising to two and duplicate side effects staying at zero.

  1. Split the workflow into deterministic orchestration and activities that perform side effects.
  2. Make each activity idempotent or keyed, because it may be retried after a partial failure.
  3. Record results in the history as each step completes, including timers and external signals.
  4. Configure retry policy per activity: attempts, backoff, and which errors are terminal.
  5. Version the workflow before changing its sequence, keeping old paths for in-flight runs.
  6. Test recovery by killing a worker mid-run and confirming the run completes without duplicate effects.

Trade-offs

ChoiceGood forCostFails when
Durable workflow engineSurvives crashes and long waitsEngine operation and programming constraintsThe team cannot adopt deterministic workflow code
Database-backed job tableSimple and familiarHand-rolled retries, leases and recoverySteps are long or human waits are common
In-process with retriesLeast infrastructureWork lost on restartRuns exceed a process lifetime
Synchronous chainEasiest to reason aboutNo recovery, timeouts cascadeAny step is slow or unreliable
Trade-off mapInteractive

Good for

Survives crashes and long waits

Cost

Engine operation and programming constraints

Fails when

The team cannot adopt deterministic workflow code

  1. The team cannot adopt deterministic workflow code
  2. Steps are long or human waits are common
  3. Runs exceed a process lifetime
  4. Any step is slow or unreliable

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Durable workflow engine is good for Survives crashes and long waits, costs Engine operation and programming constraints, and fails when The team cannot adopt deterministic workflow code. Database-backed job table is good for Simple and familiar, costs Hand-rolled retries, leases and recovery, and fails when Steps are long or human waits are common. In-process with retries is good for Least infrastructure, costs Work lost on restart, and fails when Runs exceed a process lifetime. Synchronous chain is good for Easiest to reason about, costs No recovery, timeouts cascade, and fails when Any step is slow or unreliable

Failure modes

  • Non-deterministic workflow code. Replay takes a different branch and the history mismatches. Debug: replay a recorded history in a test and assert no non-determinism error.
  • Duplicate activity. A retried activity charges a card twice. Debug: check the activity's idempotency key against the downstream record.
  • Version collision. A deploy changes the step order and in-flight runs fail to replay. Debug: run a canary with an old history fixture before rolling out workflow changes.

In production

Monitor workflow failure rate, retries per activity and the age of the oldest running workflow. Rehearse a worker kill during a run and confirm recovery completes without duplicates.

Common misconceptions

People think: Durable execution means the agent never fails.

Actually: It means the run resumes; the failing step still fails until the underlying cause changes.

People think: Any code can be made durable by wrapping it.

Actually: Workflow code must be deterministic under replay, which rules out direct clock, randomness and network calls in orchestration.

Interview angle

Strong answers separate orchestration from activities, explain replay and why determinism is required, and raise versioning for in-flight runs. Weak answers describe retries and a job table without explaining how state is reconstructed after a crash.

Sources