Why it matters
When a provider times out or a model is unavailable, the choice is between failing the request and answering worse. Answering worse is often correct, but only if the degraded answer is acceptable for that use case and if the fallback cannot itself become the overload. The failure to design against is a fallback that every request takes at once, doubling load on a dependency that is already struggling.
Mental model
It is a spare tyre: it gets you home, at reduced speed, and you are not supposed to drive on it indefinitely. The analogy breaks because a spare tyre does not create more traffic, while a fallback that retries elsewhere can amplify the load that caused the failure.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Fallback requests against offered traffic at three fallback rates, split into the requests the backup serves and those it refuses above its own 500 a minute allowance. The rate is the share of requests the primary hands over; the crossing point is the claim.
What to notice. At a 60% fallback rate a primary outage routes 720 requests a minute to a backup that serves 500, so 220 a minute are refused without any second failure. At 30% the backup carries 360 and keeps 140 a minute of headroom; its limit binds only above 1,667 offered requests, outside this chart. The fallback path needs its own quota, not the primary's.
Text version
Fallback requests per minute against offered requests per minute from 0 to 1,200, with a horizontal backup quota at 500. At a 60% fallback rate the line rises to 720 at 1,200 offered and crosses the quota at 833, marked as the point where the backup refuses requests. Two buttons select 30% and 10%: at 30% the line reaches 360 and never crosses the quota, and at 10% it reaches 120. A panel reports the fallback rate, the requests sent to the backup at full traffic and the remaining headroom.
- Classify failures by what the user can tolerate: provider outage, timeout, rate limit, malformed output, policy refusal.
- Define a ladder per class, ordered from closest quality to cheapest, and stop at the first that is acceptable.
- Bound the ladder: one hop, no recursion, and a total deadline that includes every attempt.
- Evaluate each rung on the same labelled set as the primary path, and record the quality gap.
- Mark the response as degraded with the path taken, so metrics and clients can distinguish it.
- Rate-limit the fallback independently, so it cannot exhaust the backup provider's quota.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Smaller model fallback | Keeps answering with lower quality | Two models to evaluate and operate | The task needs the larger model's reasoning |
| Cached answer fallback | Instant and cheap | Stale or mismatched answers | The request is personalised |
| Retrieval-only fallback | Grounded, no generation dependency | Poor fluency and no synthesis | Users expect a written answer |
| Fail fast with an error | Honest and simple | No answer at all | The task is low stakes and a rough answer helps |
Good for
Keeps answering with lower quality
Cost
Two models to evaluate and operate
Fails when
The task needs the larger model's reasoning
- The task needs the larger model's reasoning
- The request is personalised
- Users expect a written answer
- The task is low stakes and a rough answer helps
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Smaller model fallback is good for Keeps answering with lower quality, costs Two models to evaluate and operate, and fails when The task needs the larger model's reasoning. Cached answer fallback is good for Instant and cheap, costs Stale or mismatched answers, and fails when The request is personalised. Retrieval-only fallback is good for Grounded, no generation dependency, costs Poor fluency and no synthesis, and fails when Users expect a written answer. Fail fast with an error is good for Honest and simple, costs No answer at all, and fails when The task is low stakes and a rough answer helps
Failure modes
- Fallback storm. The primary fails, every request moves to the backup, and the backup saturates. Debug: measure fallback rate and backup utilisation together during a forced primary failure.
- Silent degradation. Users receive worse answers with no signal, and quality dashboards do not separate the paths. Debug: label responses by path and chart quality per path.
- Recursive fallback. A fallback calls a wrapper that retries the primary, and the request loops until the deadline. Debug: trace the call graph for a single request and assert a bounded depth.
In production
Monitor fallback rate, quality per path and backup quota headroom. Rehearse the switch during a game day, and alert when the fallback rate stays elevated after the primary recovers.
Common misconceptions
People think: A fallback is just a try/catch around the main call.
Actually: It is a second serving path with its own quality, capacity and failure modes, which is why it needs evaluation and a rate limit.
People think: More fallbacks mean more resilience.
Actually: Each rung adds a path to test and a way to amplify load; depth is a cost, not a virtue.
Interview angle
Separate the failure classes, define a bounded ladder and evaluate each rung. Name the load-amplification risk and the need to mark degraded responses. Describing a retry to another model as if it were free is the miss.
Sources
- AWS Builders' Library, Avoiding fallback in distributed systems — how fallback paths amplify load during failure.
- AWS Builders' Library, Timeouts, retries and backoff with jitter — deadlines and retry behaviour that bound a fallback.