Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 15 of 24

Retries, timeouts, backoff, jitter

0 of 24 complete

Why it matters

Distributed systems fail transiently. A connection resets, a load balancer drops a request during a deploy, a provider sheds load for two seconds. A single attempt turns those blips into user-visible errors; a well-behaved retry absorbs them.

The same mechanism, applied carelessly, is how a small outage becomes a large one. When a service is saturated, retries add load exactly when it can least afford it. Without jitter, every client that failed at the same moment retries at the same moment. Without a deadline, retries outlive the request that needed them. Without idempotency, they duplicate side effects. Retry logic is therefore less about persistence and more about restraint.

Mental model

A queue at a ticket counter that has just closed for a two-minute break. If everyone waits the same amount and surges forward together, the queue is worse than before. If each person waits a slightly different amount and the queue has a maximum length, service resumes smoothly.

The analogy breaks because the counter cannot tell you why it closed. A 500 might mean "try again" or "your request broke the server". You have to infer that from the status class and the operation's semantics, and you will sometimes be wrong.

How it works

Backoff ends at the budgetInteractive

One retry layer, not three. Three layers each retrying three times turn one call into 27; this figure is the layer that knows the deadline and owns the idempotency key.

Client

owns budget

Gateway

retries once

Provider

slow upstream

Step 1/5 · attempt
3 · timeout · 2000 mserror · Provider → Gateway
no response within 2000 ms
provider request id: req_9a1

The provider accepted the request and did
not answer. The attempt is retryable; the
outcome is unknown.

Step 1: attempt. Hover, tap or tab to any message to inspect its payload.

Figure. A timeout is retried with jittered backoff inside a deadline, not forever: two attempts consume 4.14 s of a 5 s budget, and the third is never started because it could not finish. The next request gets a 422 and no retry at all, since the request will stay wrong.

What to notice. The failure is a retry that outlives its caller: after the next wait, 0.49 s remain and cannot hold a 2 s attempt, so the client returns a typed error instead of starting work nobody will read. A timeout also does not prove failure, which is why every attempt repeats the idempotency key. Three retrying layers with three attempts each turn one call into 27 and are not drawn; latencies, jitter draws and ids are illustrative.

Text version

Sequence across Client, Gateway and Provider. Step 1, attempt: Client to Gateway POST /v1/generate with Idempotency-Key req_7f3c, a 5000 ms request deadline, at most 3 attempts, a 2000 ms per-attempt timeout and full jitter with a 200 ms base and 8 s cap; Gateway to Provider attempt 1 with the same key and X-Attempt 1. Step 2, timeout: Provider to Gateway no response within 2000 ms and request id req_9a1, which is retryable but leaves the outcome unknown; an in-process Gateway row classifies the timeout, computes a 0.2 s ceiling, draws 0.14 s of jitter and notes 3.00 s remaining. Step 3, try again: Gateway to Provider attempt 2 at t=2.14 s with the same key and X-Attempt 2; Provider to Gateway no response within 2000 ms and request id req_9a2 at t=4.14 s. Step 4, budget: an in-process Gateway row computes the next ceiling 0.4 s, draws 0.37 s, notes 0.86 s remaining and 0.49 s after the wait; a 2 s attempt cannot fit, so the loop stops. Gateway to Client 504 with error retry_budget_exhausted, attempts 2, elapsed_ms 4140, retryable true and the request id. Step 5, stop: Client to Gateway a second request whose max_output_tokens is the string 512; Gateway to Client 422 Unprocessable Content with a pointer at /max_output_tokens; an in-process Gateway row records the policy: 422 is not retried, 408 and 429 are, 5xx retry inside the deadline, and a POST retry needs the idempotency key.

  1. Classify the failure. Connection refused, reset or timeout before any response: usually retryable. 408, 429, 500, 502, 503, 504: usually retryable. 400, 401, 403, 404, 409, 422: not retryable — the request is wrong and will stay wrong.
  2. Check whether the operation is repeatable. Reads are safe. PUT and DELETE are idempotent by definition. POST is not, unless the server supports an idempotency key. A retried POST without one is a duplicate waiting to happen.
  3. Set a per-attempt timeout and a total deadline. The timeout bounds one attempt; the deadline bounds the whole operation including waits. Never let the sum of allowed attempts exceed the caller's remaining budget — pass the remaining time down.
  4. Wait exponentially. Delay grows with the attempt number, capped at a maximum: da=min⁡(dmax,d0⋅2a)d_a = \min(d_{max}, d_0 \cdot 2^a) for attempt aa. The cap stops one unlucky request from waiting minutes.
  5. Add jitter. Replace the fixed delay with a random value in a range. Full jitter draws uniformly from [0,da][0, d_a]; equal jitter uses da/2+U(0,da/2)d_a/2 + U(0, d_a/2) to keep a floor. Jitter converts a synchronised herd into a spread.
  6. Cap the total attempts, then fail. A retry budget is a design decision, not an infinite loop. When attempts are exhausted, return a typed error the caller can handle, not the last raw exception.
  7. Add a circuit breaker for persistent failure. After a threshold of consecutive failures, stop sending for a cool-down period and fail fast. This protects both sides: the dependency gets room to recover, and your latency stays bounded.
  8. Retry at one layer. Ideally the layer that knows whether the operation is idempotent and how much budget remains. Nested retries multiply.

Retry with a deadline

import asyncio
import random
 
def backoff_delay(attempt: int, base: float = 0.2, cap: float = 8.0) -> float:
    ceiling = min(cap, base * 2 ** attempt)
    return random.uniform(0, ceiling)          # full jitter
 
async def call_with_retry(fn, *, attempts: int = 4, deadline: float = 15.0,
                          per_attempt: float = 5.0):
    loop = asyncio.get_running_loop()
    end = loop.time() + deadline
    for attempt in range(attempts):
        try:
            remaining = end - loop.time()
            if remaining <= 0:
                raise TimeoutError("retry deadline exceeded")
            # Per-attempt timeout, capped by the remaining deadline.
            return await asyncio.wait_for(fn(), timeout=min(per_attempt, remaining))
        except (RetryableError, asyncio.TimeoutError):   # 408/429/5xx and stuck attempts
            remaining = end - loop.time()
            delay = backoff_delay(attempt)
            if attempt == attempts - 1 or delay >= remaining:
                raise
            await asyncio.sleep(delay)

The two guards that matter are delay >= remaining and the deadline passed into wait_for. Without them the retry loop can outlive the request, and the caller has already given up while you keep consuming capacity.

againfatalFailureClassifyJittered backoffRetryDeadline
Step 1/4 · Classify the failure
Figure. Retries help with transient faults and make overload worse; classify before you retry.
Text version

A failure is classified; retryable failures wait with jittered backoff and retry; fatal failures stop; every path respects the deadline.

Trade-offs

StrategyGood forCostFails when
No retryNon-idempotent writes, interactive deadlinesAbsorbs no transient failureThe dependency has routine blips
Fixed-delay retrySimple internal calls, single clientSynchronised retries under loadMany clients fail at once
Exponential backoff, no jitterPredictable pacing in testsThundering herd on recoveryTraffic is concurrent and bursty
Exponential backoff with jitterMost network and model callsSlightly longer tail latencyThe deadline is shorter than the first backoff
Circuit breakerPersistently failing dependenciesFast failures users must handleThe failure is per-tenant, not global
Queue and drainWork that can wait, batch jobsExtra infrastructure and lagThe user is waiting on the result
Trade-off mapInteractive

Good for

Non-idempotent writes, interactive deadlines

Cost

Absorbs no transient failure

Fails when

The dependency has routine blips

  1. The dependency has routine blips
  2. Many clients fail at once
  3. Traffic is concurrent and bursty
  4. The deadline is shorter than the first backoff
  5. The failure is per-tenant, not global

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: No retry is good for Non-idempotent writes, interactive deadlines, costs Absorbs no transient failure, and fails when The dependency has routine blips. Fixed-delay retry is good for Simple internal calls, single client, costs Synchronised retries under load, and fails when Many clients fail at once. Exponential backoff, no jitter is good for Predictable pacing in tests, costs Thundering herd on recovery, and fails when Traffic is concurrent and bursty. Exponential backoff with jitter is good for Most network and model calls, costs Slightly longer tail latency, and fails when The deadline is shorter than the first backoff. Circuit breaker is good for Persistently failing dependencies, costs Fast failures users must handle, and fails when The failure is per-tenant, not global

Failure modes

Retry amplification. Three layers each retry three times, so one failure becomes twenty-seven requests. Debug: count upstream requests per user action during an injected error; anything above the configured attempt count means duplicated retry logic.

Duplicate side effects. A POST that sends an email is retried after a read timeout, and the user receives two. Debug: inspect whether every retried write carries an idempotency key, then check the server's dedupe store.

Retrying past the deadline. The client gives up at ten seconds while the server keeps retrying for another thirty. Debug: compare client and server attempt counts for the same trace ID; a mismatch means deadlines are not propagated.

Retrying non-retryable errors. A schema validation error is retried four times, adding latency and no value. Debug: break down retry counts by status code and error class; any 4xx outside 408 and 429 is a policy bug.

In production

Two controls keep retries honest. A retry budget caps retries as a fraction of successful traffic, so a retry loop cannot exceed the capacity freed by healthy requests — gRPC's client retry design calls this retry throttling, and it is worth copying even in a custom client. A circuit breaker converts a decaying dependency into fast, predictable failures with a recovery probe. Instrument attempts per request, retry success rate, time spent waiting, and the age of the oldest in-flight request. If retries are working, retry success rate is high and added latency is small; if they are failing, you will see attempt counts climb while success stays flat, which is the moment to stop retrying and start shedding.

Common misconceptions

People think: Retries improve reliability, so more retries are better.

Actually: Retries improve reliability only when the failure is transient and capacity exists. Under saturation they convert a partial outage into a total one.

People think: A timeout means the request did not happen.

Actually: It means you did not receive a response in time. The server may have completed the work, which is why retrying a write requires idempotency.

People think: Jitter is a minor optimisation.

Actually: Jitter is the difference between a synchronised retry wave and a spread. Without it, clients that fail together recover together and re-overload the service.

Interview angle

"Design a client for a model API that occasionally returns 503" is a compact test of several instincts at once. Cover the failure classification, the idempotency requirement, the timeout hierarchy, exponential backoff with jitter, the retry budget, and what the user sees after the final attempt. Then say what you would monitor. Candidates who only add a decorator tend to miss the amplification question that follows.

See production questions and idempotency.

Sources