Why it matters
Retries are mandatory. Networks drop responses, load balancers restart, clients time out after the server has already committed. Every one of those cases leaves the caller unsure whether the work happened, and the safe response is to ask again.
Without idempotency, "ask again" is a new operation. The customer is charged twice, two onboarding emails arrive, two tickets are filed, the inventory decrements twice. This is not an edge case at scale: it is a certainty, and it is the reason payment APIs expose idempotency keys as a first-class feature.
The problem gets sharper with agents. A model-driven loop may call the same tool repeatedly after a failed observation — resend the email, re-open the issue, re-run the deployment. The loop cannot know whether the first call landed, so the tool has to make the answer safe.
Mental model
A coat check ticket. Hand over the same ticket twice and you get the same coat, not a second coat. The ticket identifies an operation that has already happened, and the counter's records decide the answer.
The analogy breaks under concurrency. Two people can present the same ticket at two windows at the same instant. A real coat check relies on one attendant and one rack. Software needs an atomic claim — a uniqueness constraint or compare-and-set — or both requests will see "not found" and both will fetch a coat.
How it works
- 1The client creates a key per logical operation. One key for "submit this order", reused unchanged across every retry of that submission. A UUID generated inside the retry loop defeats the whole design.
- 2The server scopes the key. Keys are namespaced by account and endpoint, so one tenant cannot collide with or probe another's keys. Scope is part of the primary key.
- 3Claim the key atomically, before the side effect. Insert the key with a unique constraint and record the request fingerprint. If the insert conflicts, this is a duplicate and you handle it explicitly. The claim must commit before the external call, otherwise a crash between the effect and the record leaves no trace.
- 4On duplicate, distinguish three states. Completed: replay the stored status and body. In flight: return a conflict or wait briefly, never execute again. Fingerprint mismatch: reject, because the same key with a different body is a client bug, not a retry.
- 5Store enough to replay. Response status, body and any identifiers the caller needs. Replaying a 201 with the original resource ID is what makes the retry transparent.
- 6Make the underlying effect idempotent too. A key store protects the API boundary; the database write still needs an upsert or a natural key, and the downstream provider may need its own key. Defence in depth, because the key store can fail after the effect succeeds.
Step 1 of 6
The client creates a key per logical operation. One key for "submit this order", reused unchanged across every retry of that submission. A UUID generated inside the retry loop defeats the whole design.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. The client creates a key per logical operation. One key for "submit this order", reused u… 2. The server scopes the key. Keys are namespaced by account and endpoint, so one tenant can… 3. Claim the key atomically, before the side effect. Insert the key with a unique constraint… 4. On duplicate, distinguish three states. Completed: replay the stored status and body. In … 5. Store enough to replay. Response status, body and any identifiers the caller needs. Repla… 6. Make the underlying effect idempotent too. A key store protects the API boundary; the dat…
- The client creates a key per logical operation. One key for "submit this order", reused unchanged across every retry of that submission. A UUID generated inside the retry loop defeats the whole design.
- The server scopes the key. Keys are namespaced by account and endpoint, so one tenant cannot collide with or probe another's keys. Scope is part of the primary key.
- Claim the key atomically, before the side effect. Insert the key with a unique constraint and record the request fingerprint. If the insert conflicts, this is a duplicate and you handle it explicitly. The claim must commit before the external call, otherwise a crash between the effect and the record leaves no trace.
- On duplicate, distinguish three states. Completed: replay the stored status and body. In flight: return a conflict or wait briefly, never execute again. Fingerprint mismatch: reject, because the same key with a different body is a client bug, not a retry.
- Store enough to replay. Response status, body and any identifiers the caller needs. Replaying a
201with the original resource ID is what makes the retry transparent. - Make the underlying effect idempotent too. A key store protects the API boundary; the database write still needs an upsert or a natural key, and the downstream provider may need its own key. Defence in depth, because the key store can fail after the effect succeeds.
- Expire keys deliberately. Retention must cover the longest plausible retry horizon, including a client that goes offline and resumes. After that, the key is garbage, but the effect must still be idempotent by construction — for example an upsert keyed on a business identifier.
- Reconcile when the record is incomplete. If a process dies between the effect and the response write, a background job or a status endpoint resolves the ambiguity rather than leaving the caller to guess.
The store
CREATE TABLE idempotency_keys (
scope text NOT NULL,
key text NOT NULL,
request_hash text NOT NULL,
state text NOT NULL CHECK (state IN ('in_flight', 'done')),
response_code integer,
response_body jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (scope, key)
);
-- Claim the key. Zero rows returned means a duplicate.
INSERT INTO idempotency_keys (scope, key, request_hash, state)
VALUES ($1, $2, $3, 'in_flight')
ON CONFLICT (scope, key) DO NOTHING
RETURNING key;The primary key is the concurrency control: one transaction wins the insert, the other gets no row and takes the duplicate path. A read-then-write check without a constraint lets both requests pass the check.
- Idempotent by definition
- GET, HEAD, PUT, DELETE, OPTIONS, TRACE
- Needs a key
- POST endpoints that create or charge
- Correctness primitive
- Unique constraint on (scope, key)
- Three duplicate states
- Completed, in flight, fingerprint mismatch
- Retention rule
- Longer than the longest retry horizon
Trade-offs
| Approach | Good for | Cost | Fails when |
|---|---|---|---|
| Natural key upsert | Entities with a stable business identifier | Requires a real natural key | Identifiers are generated per request |
| Idempotency-key store | Payments, orders, external side effects | A store, a TTL policy and duplicate handling | Keys are generated per attempt or never expire |
| Request-hash dedupe | Internal jobs with deterministic payloads | False positives on legitimately repeated requests | Two identical requests are both intended |
| At-least-once plus reconciliation | Batch pipelines, eventual consistency | Delayed correctness, extra job | The user waits for an immediate answer |
| No dedupe, manual cleanup | Prototypes | Double effects and support load | Money or email is involved |
Good for
Entities with a stable business identifier
Cost
Requires a real natural key
Fails when
Identifiers are generated per request
- Identifiers are generated per request
- Keys are generated per attempt or never expire
- Two identical requests are both intended
- The user waits for an immediate answer
- Money or email is involved
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Natural key upsert is good for Entities with a stable business identifier, costs Requires a real natural key, and fails when Identifiers are generated per request. Idempotency-key store is good for Payments, orders, external side effects, costs A store, a TTL policy and duplicate handling, and fails when Keys are generated per attempt or never expire. Request-hash dedupe is good for Internal jobs with deterministic payloads, costs False positives on legitimately repeated requests, and fails when Two identical requests are both intended. At-least-once plus reconciliation is good for Batch pipelines, eventual consistency, costs Delayed correctness, extra job, and fails when The user waits for an immediate answer. No dedupe, manual cleanup is good for Prototypes, costs Double effects and support load, and fails when Money or email is involved
Failure modes
Key generated inside the retry loop. Every attempt is unique and every duplicate executes. Debug: assert one key per logical operation in the client test, not one per HTTP call.
Read-then-write race. Two concurrent duplicates both find no record and both proceed. Debug: run the same request twice in parallel against staging and count the side effects; a unique constraint is the only reliable fix.
Fingerprint ignored. A client reuses a key for a different payload and receives the first response, silently dropping the second request. Debug: store a hash of the canonicalised body and return a distinct error on mismatch.
Unbounded retention. The key table grows without limit and becomes the largest table in the database. Debug: check row counts and index size monthly; partition by creation date and expire on a schedule.
In production
Monitor duplicate rate, replay rate, fingerprint mismatches and in-flight conflicts. A sudden rise in duplicates usually means a client-side timeout is set too low — the server is fine, the caller gives up early. Keep the key store on the same transactional boundary as the primary write where possible; if the effect is external, pair the store with a reconciliation job that queries the provider for the operation's final state. For agent tools, apply the same discipline per tool call: an email tool needs a dedupe key, a ticket tool needs a natural key, and a read-only tool needs none.
Common misconceptions
People think: Idempotency means the response is identical.
Actually: It means the intended effect is identical. A replayed response is a convenience; the essential guarantee is that the second request does not apply the change twice.
People think: A database transaction gives me idempotency across an external API call.
Actually: The transaction cannot roll back a charge or an email that already left the system. You need a recorded key plus an idempotent or reconcilable external effect.
People think: Exactly-once delivery solves this.
Actually: Exactly-once delivery is not achievable end to end over a network that can lose acknowledgements. Systems achieve effectively-once processing through at-least-once delivery plus deduplication.
Interview angle
The question arrives as "how do you prevent a double charge when the client retries?" and the follow-ups test whether you understand the ordering. Answer with the key lifecycle, the atomic claim before the effect, the three duplicate states, and retention. Then mention the failure that remains: a crash after the effect and before the record, and how reconciliation covers it. That last part is what distinguishes a design from a diagram.
See production questions and retries, timeouts and jitter.
Sources
- RFC 9110, section 9.2.2: Idempotent methods — the normative definition and which methods qualify.
- Stripe: idempotent requests — a production key-based design, including replay behaviour and key retention.
- PostgreSQL: constraints — unique and primary-key constraints as the atomic claim primitive.