Why it matters
An agent retries a timed-out call, a queue redelivers a message, a resumed run replays the step that was in flight. Each is correct at the transport layer and a duplicate charge at the business layer. Making tools idempotent moves the problem from hoping retries do not happen to making a second call harmless, which is the only assumption that survives real infrastructure.
Mental model
An idempotency key is a claim ticket: whoever holds the same ticket gets the same coat, not a second one. The metaphor breaks because a cloakroom has one attendant and one queue, while a distributed system has concurrent requests carrying the same key arriving at different replicas at once.
How it works
Keyed transitions
- Execute → Torn commit: key write lost
Event · step 1 of 9
Call with key
Entry: attempt 1 of charge_card for order 8412. The key is derived from the step, step4-8412, not generated per attempt.
In state
- Call
Next: Lookup misses
- Attempts
- 1
- External effects
- 0
Figure. Eight states. The key is derived from the step, claimed with a uniqueness constraint before the effect, and the result is committed with it. A hit at Lookup replays the first result; a key per attempt and a torn commit are the two ways the effect runs twice.
What to notice. The claim removes the check-then-act window: the second concurrent request loses the insert and waits for the first result. The dedupe record is written in the same transaction as the effect, and its retention must outlive the maximum retry and resume horizon. Key format, the order id, the 24-hour retention window and the attempt counts are illustrative, and provider-side idempotency is drawn as the tool's own behaviour.
Text version
A state machine with eight states. Call goes to Lookup on the step key, and to Duplicate charge when the key is per attempt. Lookup goes to Claim when the key is absent, and to Done when it is present. Claim goes to Execute when the insert wins, and to Done when it loses the insert and waits for the stored result. Execute goes to Commit on the effect, and to Torn commit when the key write is lost after the effect. Commit goes to Done on the stored result. Torn commit returns to Claim on retry. The trace walks Call, Lookup, Claim, Execute, Commit, Done, then a replay, then Duplicate charge, then Torn commit. Counters show attempts and external effects.
- Decide the natural key for each tool: order id, message id, or a hash of the intended effect.
- Require the caller to supply the key, derived from the task step rather than randomised per attempt.
- Store the key with the result in the same transaction as the effect.
- On a duplicate, return the stored result instead of re-executing, including the original error.
- Set a retention window longer than the maximum retry and resume horizon.
- Handle concurrent duplicates with a uniqueness constraint rather than a read-then-write check.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Natural key | No extra state, obvious semantics | Only exists for some operations | The operation creates a new entity each time |
| Caller-supplied key | Works for any write | Callers can reuse or randomise keys | Multiple callers write the same resource |
| Dedupe table with expiry | Simple and general | Storage and a retention decision | Retries arrive after the window |
| Attempt at exactly-once | No duplicate handling | Not achievable across independent systems | Any network boundary is involved |
Good for
No extra state, obvious semantics
Cost
Only exists for some operations
Fails when
The operation creates a new entity each time
- The operation creates a new entity each time
- Multiple callers write the same resource
- Retries arrive after the window
- Any network boundary is involved
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Natural key is good for No extra state, obvious semantics, costs Only exists for some operations, and fails when The operation creates a new entity each time. Caller-supplied key is good for Works for any write, costs Callers can reuse or randomise keys, and fails when Multiple callers write the same resource. Dedupe table with expiry is good for Simple and general, costs Storage and a retention decision, and fails when Retries arrive after the window. Attempt at exactly-once is good for No duplicate handling, costs Not achievable across independent systems, and fails when Any network boundary is involved
Failure modes
- Key per attempt. Every retry has a new key and duplicates the effect. Debug: assert the key is derived from the step, and count distinct keys per step in the logs.
- Non-atomic dedupe. The effect commits and the key write fails, so a retry repeats it. Debug: inject a crash between the two writes and check for duplicates.
- Window too short. A late retry after the retention period creates a second record. Debug: compare the dedupe window with the maximum resume age.
In production
Monitor duplicate-key hits, dedupe table growth and the age of the oldest key. Alert when a tool's duplicate rate rises, which usually means a timeout or retry policy changed.
Common misconceptions
People think: A retry only happens when something is broken.
Actually: Timeouts, deploys and resumes cause retries on a healthy system, so the tool must tolerate them.
People think: A unique database index makes any tool idempotent.
Actually: It prevents duplicate rows, not duplicate emails, charges or external calls.
Interview angle
Strong answers derive the key from intent, write it with the effect in one transaction, and choose a retention window from the retry horizon. They mention the concurrent-duplicate case. Weak answers say the endpoint is safe because it uses PUT.
Sources
- AWS Builders' Library, Making retries safe with idempotent APIs — idempotency keys and dedupe stores for retried writes.
- RFC 9110, HTTP Semantics — the definition of idempotent methods and their limits.
- Stripe API reference, Idempotent requests — a production idempotency-key contract with replay semantics.