Why it matters
A model call is a network dependency with a quota, and the provider's capacity is shared with every other customer. Under load it returns throttling errors, slower responses, or both. Applications that treat it like a local function fail in ways users notice: retry storms, timeouts that stack, and fallbacks that have never been exercised. Reliability work moves those failures into a controlled path with budgets and rehearsed degradation.
Mental model
Provider reliability is driving in traffic: you cannot make the road faster, only leave earlier and know a second route. The metaphor breaks because the traffic report is written by the road operator and arrives late, so you often learn about congestion from your own latency.
How it works
- 1Classify every failure the client can see: timeout, connection error, throttling, server error, invalid request and content block.
- 2Retry only retryable classes, with exponential backoff, jitter and a per-request deadline that bounds total attempts.
- 3Use a shared token bucket per provider and per tenant so retries cannot exceed the quota.
- 4Define a degraded path such as a smaller model, a cached response or a reduced feature, and route to it on sustained errors.
- 5Exercise failover in staging with forced errors and measure the quality of the fallback on the same evaluation set.
- 6Track error rate, retry rate and p95 latency per provider, and alert on changes in the retry-to-success ratio.
Step 1 of 6
Classify every failure the client can see: timeout, connection error, throttling, server error, invalid request and content block.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Classify every failure the client can see: timeout, connection error, throttling, server … 2. Retry only retryable classes, with exponential backoff, jitter and a per-request deadline… 3. Use a shared token bucket per provider and per tenant so retries cannot exceed the quota. 4. Define a degraded path such as a smaller model, a cached response or a reduced feature, a… 5. Exercise failover in staging with forced errors and measure the quality of the fallback o… 6. Track error rate, retry rate and p95 latency per provider, and alert on changes in the re…
- Classify every failure the client can see: timeout, connection error, throttling, server error, invalid request and content block.
- Retry only retryable classes, with exponential backoff, jitter and a per-request deadline that bounds total attempts.
- Use a shared token bucket per provider and per tenant so retries cannot exceed the quota.
- Define a degraded path such as a smaller model, a cached response or a reduced feature, and route to it on sustained errors.
- Exercise failover in staging with forced errors and measure the quality of the fallback on the same evaluation set.
- Track error rate, retry rate and p95 latency per provider, and alert on changes in the retry-to-success ratio.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Aggressive retries | Hides transient errors | Amplifies throttling and adds latency | The provider is rate limiting rather than failing |
| Immediate failover | Fast recovery | Fallback quality may be lower and untested | The primary recovers in seconds |
| Circuit breaker | Stops cascading failure | Rejects requests while the probe is open | Traffic is bursty and short |
| Queue and wait | Rides out short spikes | Latency grows unbounded | The deadline is interactive |
Good for
Hides transient errors
Cost
Amplifies throttling and adds latency
Fails when
The provider is rate limiting rather than failing
- The provider is rate limiting rather than failing
- The primary recovers in seconds
- Traffic is bursty and short
- The deadline is interactive
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Aggressive retries is good for Hides transient errors, costs Amplifies throttling and adds latency, and fails when The provider is rate limiting rather than failing. Immediate failover is good for Fast recovery, costs Fallback quality may be lower and untested, and fails when The primary recovers in seconds. Circuit breaker is good for Stops cascading failure, costs Rejects requests while the probe is open, and fails when Traffic is bursty and short. Queue and wait is good for Rides out short spikes, costs Latency grows unbounded, and fails when The deadline is interactive
Failure modes
- Retry storm. All clients retry at once and the provider throttles harder. Debug: compare retry rate with error rate and check for jitter in the backoff distribution.
- Deadline stacking. Each layer applies its own timeout and the total exceeds the user's budget. Debug: trace the request across layers and sum the configured timeouts.
- Cold fallback. The secondary provider fails on first real use because its prompt or model was never tested. Debug: force the fallback in staging weekly and score its output.
In production
Track success rate within deadline, not just error rate, and rehearse failover on a schedule. Cap total retries per request and alert when the fallback carries more than a small share of traffic.
Common misconceptions
People think: Provider SLAs remove the need for retries.
Actually: An SLA describes availability over a period, not the outcome of your request at this second.
People think: Failover is a configuration flag.
Actually: It is a second integration with its own prompts, limits and quality, and it needs its own evaluation.
Interview angle
Classify failures, then bound the response with deadlines, jittered retries and a shared quota. Strong answers include a rehearsed degraded path and per-tenant fairness. Weak answers add a retry loop and call it resilience, without saying what happens when the fallback is also throttled.
Sources
- AWS Builders' Library, Timeouts, retries and backoff with jitter — why jitter and a retry budget are necessary.
- OpenAI, Rate limits guide — quota dimensions and throttling behaviour.
- Google SRE Book, Handling overload — load shedding and client-side throttling.