Why it matters
Distributed systems fail transiently. A connection resets, a load balancer drops a request during a deploy, a provider sheds load for two seconds. A single attempt turns those blips into user-visible errors; a well-behaved retry absorbs them.
The same mechanism, applied carelessly, is how a small outage becomes a large one. When a service is saturated, retries add load exactly when it can least afford it. Without jitter, every client that failed at the same moment retries at the same moment. Without a deadline, retries outlive the request that needed them. Without idempotency, they duplicate side effects. Retry logic is therefore less about persistence and more about restraint.
Mental model
A queue at a ticket counter that has just closed for a two-minute break. If everyone waits the same amount and surges forward together, the queue is worse than before. If each person waits a slightly different amount and the queue has a maximum length, service resumes smoothly.
The analogy breaks because the counter cannot tell you why it closed. A 500 might mean "try again" or "your request broke the server". You have to infer that from the status class and the operation's semantics, and you will sometimes be wrong.
How it works
One retry layer, not three. Three layers each retrying three times turn one call into 27; this figure is the layer that knows the deadline and owns the idempotency key.
Client
owns budget
Gateway
retries once
Provider
slow upstream
no response within 2000 ms
provider request id: req_9a1
The provider accepted the request and did
not answer. The attempt is retryable; the
outcome is unknown.Step 1: attempt. Hover, tap or tab to any message to inspect its payload.
Figure. A timeout is retried with jittered backoff inside a deadline, not forever: two attempts consume 4.14 s of a 5 s budget, and the third is never started because it could not finish. The next request gets a 422 and no retry at all, since the request will stay wrong.
What to notice. The failure is a retry that outlives its caller: after the next wait, 0.49 s remain and cannot hold a 2 s attempt, so the client returns a typed error instead of starting work nobody will read. A timeout also does not prove failure, which is why every attempt repeats the idempotency key. Three retrying layers with three attempts each turn one call into 27 and are not drawn; latencies, jitter draws and ids are illustrative.
Text version
Sequence across Client, Gateway and Provider. Step 1, attempt: Client to Gateway POST /v1/generate with Idempotency-Key req_7f3c, a 5000 ms request deadline, at most 3 attempts, a 2000 ms per-attempt timeout and full jitter with a 200 ms base and 8 s cap; Gateway to Provider attempt 1 with the same key and X-Attempt 1. Step 2, timeout: Provider to Gateway no response within 2000 ms and request id req_9a1, which is retryable but leaves the outcome unknown; an in-process Gateway row classifies the timeout, computes a 0.2 s ceiling, draws 0.14 s of jitter and notes 3.00 s remaining. Step 3, try again: Gateway to Provider attempt 2 at t=2.14 s with the same key and X-Attempt 2; Provider to Gateway no response within 2000 ms and request id req_9a2 at t=4.14 s. Step 4, budget: an in-process Gateway row computes the next ceiling 0.4 s, draws 0.37 s, notes 0.86 s remaining and 0.49 s after the wait; a 2 s attempt cannot fit, so the loop stops. Gateway to Client 504 with error retry_budget_exhausted, attempts 2, elapsed_ms 4140, retryable true and the request id. Step 5, stop: Client to Gateway a second request whose max_output_tokens is the string 512; Gateway to Client 422 Unprocessable Content with a pointer at /max_output_tokens; an in-process Gateway row records the policy: 422 is not retried, 408 and 429 are, 5xx retry inside the deadline, and a POST retry needs the idempotency key.
- Classify the failure. Connection refused, reset or timeout before any response: usually retryable.
408,429,500,502,503,504: usually retryable.400,401,403,404,409,422: not retryable — the request is wrong and will stay wrong. - Check whether the operation is repeatable. Reads are safe.
PUTandDELETEare idempotent by definition.POSTis not, unless the server supports an idempotency key. A retriedPOSTwithout one is a duplicate waiting to happen. - Set a per-attempt timeout and a total deadline. The timeout bounds one attempt; the deadline bounds the whole operation including waits. Never let the sum of allowed attempts exceed the caller's remaining budget — pass the remaining time down.
- Wait exponentially. Delay grows with the attempt number, capped at a maximum: for attempt . The cap stops one unlucky request from waiting minutes.
- Add jitter. Replace the fixed delay with a random value in a range. Full jitter draws uniformly from ; equal jitter uses to keep a floor. Jitter converts a synchronised herd into a spread.
- Cap the total attempts, then fail. A retry budget is a design decision, not an infinite loop. When attempts are exhausted, return a typed error the caller can handle, not the last raw exception.
- Add a circuit breaker for persistent failure. After a threshold of consecutive failures, stop sending for a cool-down period and fail fast. This protects both sides: the dependency gets room to recover, and your latency stays bounded.
- Retry at one layer. Ideally the layer that knows whether the operation is idempotent and how much budget remains. Nested retries multiply.
Retry with a deadline
import asyncio
import random
def backoff_delay(attempt: int, base: float = 0.2, cap: float = 8.0) -> float:
ceiling = min(cap, base * 2 ** attempt)
return random.uniform(0, ceiling) # full jitter
async def call_with_retry(fn, *, attempts: int = 4, deadline: float = 15.0,
per_attempt: float = 5.0):
loop = asyncio.get_running_loop()
end = loop.time() + deadline
for attempt in range(attempts):
try:
remaining = end - loop.time()
if remaining <= 0:
raise TimeoutError("retry deadline exceeded")
# Per-attempt timeout, capped by the remaining deadline.
return await asyncio.wait_for(fn(), timeout=min(per_attempt, remaining))
except (RetryableError, asyncio.TimeoutError): # 408/429/5xx and stuck attempts
remaining = end - loop.time()
delay = backoff_delay(attempt)
if attempt == attempts - 1 or delay >= remaining:
raise
await asyncio.sleep(delay)The two guards that matter are delay >= remaining and the deadline passed into wait_for. Without them the retry loop can outlive the request, and the caller has already given up while you keep consuming capacity.
Text version
A failure is classified; retryable failures wait with jittered backoff and retry; fatal failures stop; every path respects the deadline.
Trade-offs
| Strategy | Good for | Cost | Fails when |
|---|---|---|---|
| No retry | Non-idempotent writes, interactive deadlines | Absorbs no transient failure | The dependency has routine blips |
| Fixed-delay retry | Simple internal calls, single client | Synchronised retries under load | Many clients fail at once |
| Exponential backoff, no jitter | Predictable pacing in tests | Thundering herd on recovery | Traffic is concurrent and bursty |
| Exponential backoff with jitter | Most network and model calls | Slightly longer tail latency | The deadline is shorter than the first backoff |
| Circuit breaker | Persistently failing dependencies | Fast failures users must handle | The failure is per-tenant, not global |
| Queue and drain | Work that can wait, batch jobs | Extra infrastructure and lag | The user is waiting on the result |
Good for
Non-idempotent writes, interactive deadlines
Cost
Absorbs no transient failure
Fails when
The dependency has routine blips
- The dependency has routine blips
- Many clients fail at once
- Traffic is concurrent and bursty
- The deadline is shorter than the first backoff
- The failure is per-tenant, not global
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: No retry is good for Non-idempotent writes, interactive deadlines, costs Absorbs no transient failure, and fails when The dependency has routine blips. Fixed-delay retry is good for Simple internal calls, single client, costs Synchronised retries under load, and fails when Many clients fail at once. Exponential backoff, no jitter is good for Predictable pacing in tests, costs Thundering herd on recovery, and fails when Traffic is concurrent and bursty. Exponential backoff with jitter is good for Most network and model calls, costs Slightly longer tail latency, and fails when The deadline is shorter than the first backoff. Circuit breaker is good for Persistently failing dependencies, costs Fast failures users must handle, and fails when The failure is per-tenant, not global
Failure modes
Retry amplification. Three layers each retry three times, so one failure becomes twenty-seven requests. Debug: count upstream requests per user action during an injected error; anything above the configured attempt count means duplicated retry logic.
Duplicate side effects. A POST that sends an email is retried after a read timeout, and the user receives two. Debug: inspect whether every retried write carries an idempotency key, then check the server's dedupe store.
Retrying past the deadline. The client gives up at ten seconds while the server keeps retrying for another thirty. Debug: compare client and server attempt counts for the same trace ID; a mismatch means deadlines are not propagated.
Retrying non-retryable errors. A schema validation error is retried four times, adding latency and no value. Debug: break down retry counts by status code and error class; any 4xx outside 408 and 429 is a policy bug.
In production
Two controls keep retries honest. A retry budget caps retries as a fraction of successful traffic, so a retry loop cannot exceed the capacity freed by healthy requests — gRPC's client retry design calls this retry throttling, and it is worth copying even in a custom client. A circuit breaker converts a decaying dependency into fast, predictable failures with a recovery probe. Instrument attempts per request, retry success rate, time spent waiting, and the age of the oldest in-flight request. If retries are working, retry success rate is high and added latency is small; if they are failing, you will see attempt counts climb while success stays flat, which is the moment to stop retrying and start shedding.
Common misconceptions
People think: Retries improve reliability, so more retries are better.
Actually: Retries improve reliability only when the failure is transient and capacity exists. Under saturation they convert a partial outage into a total one.
People think: A timeout means the request did not happen.
Actually: It means you did not receive a response in time. The server may have completed the work, which is why retrying a write requires idempotency.
People think: Jitter is a minor optimisation.
Actually: Jitter is the difference between a synchronised retry wave and a spread. Without it, clients that fail together recover together and re-overload the service.
Interview angle
"Design a client for a model API that occasionally returns 503" is a compact test of several instincts at once. Cover the failure classification, the idempotency requirement, the timeout hierarchy, exponential backoff with jitter, the retry budget, and what the user sees after the final attempt. Then say what you would monitor. Candidates who only add a decorator tend to miss the amplification question that follows.
See production questions and idempotency.
Sources
- Timeouts, retries and backoff with jitter — the failure taxonomy, retry amplification and why jitter is required at scale.
- Exponential backoff and jitter — the full and equal jitter algorithms and the measured effect on contention.
- gRPC client retries (A6) — a concrete retry policy shape with maximum attempts, backoff parameters and retry throttling.
- Addressing cascading failures — Google SRE on how retries and queueing turn a small failure into a system-wide one.