Why it matters
Failover involves more than a conditional in the client: credentials must exist for the backup, quotas must be raised, DNS or routing must point elsewhere, and the fallback model must produce acceptable answers. Each is a separate failure point that only appears when the primary is actually gone. Testing converts the unknown into a measured switch time and a list of things that did not work.
Mental model
It is a fire drill: the point is not the fire, it is discovering that the stairwell door is locked. The analogy breaks because a drill can be scheduled at a convenient time, while the real failure arrives at peak traffic with the on-call engineer asleep.
How it works
- 1Enumerate the failure scenarios: provider outage, region loss, quota exhaustion, model deprecation, credential expiry.
- 2For each, write the expected detection signal, the switch mechanism and the target recovery time.
- 3Run the scenario in staging first, then in production with a bounded blast radius such as one tenant or one route.
- 4Time detection and switch separately, and record where the delay came from.
- 5Check the standby's capacity and quality under the inherited load, not just that it responds.
- 6Update the runbook with what actually happened, including the commands that were needed and missing.
Step 1 of 6
Enumerate the failure scenarios: provider outage, region loss, quota exhaustion, model deprecation, credential expiry.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Enumerate the failure scenarios: provider outage, region loss, quota exhaustion, model de… 2. For each, write the expected detection signal, the switch mechanism and the target recove… 3. Run the scenario in staging first, then in production with a bounded blast radius such as… 4. Time detection and switch separately, and record where the delay came from. 5. Check the standby's capacity and quality under the inherited load, not just that it respo… 6. Update the runbook with what actually happened, including the commands that were needed a…
- Enumerate the failure scenarios: provider outage, region loss, quota exhaustion, model deprecation, credential expiry.
- For each, write the expected detection signal, the switch mechanism and the target recovery time.
- Run the scenario in staging first, then in production with a bounded blast radius such as one tenant or one route.
- Time detection and switch separately, and record where the delay came from.
- Check the standby's capacity and quality under the inherited load, not just that it responds.
- Update the runbook with what actually happened, including the commands that were needed and missing.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Staging-only drills | No customer risk | Misses production credentials, DNS and quotas | The failure involves infrastructure configuration |
| Production with small blast radius | Realistic and bounded | Some users affected | The route cannot be isolated |
| Full production failover | Proves end-to-end capacity | Large risk and coordination cost | The system is fragile or the team is small |
| Automated chaos experiments | Continuous coverage | Needs guardrails and observability | Nobody reviews the results |
Good for
No customer risk
Cost
Misses production credentials, DNS and quotas
Fails when
The failure involves infrastructure configuration
- The failure involves infrastructure configuration
- The route cannot be isolated
- The system is fragile or the team is small
- Nobody reviews the results
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Staging-only drills is good for No customer risk, costs Misses production credentials, DNS and quotas, and fails when The failure involves infrastructure configuration. Production with small blast radius is good for Realistic and bounded, costs Some users affected, and fails when The route cannot be isolated. Full production failover is good for Proves end-to-end capacity, costs Large risk and coordination cost, and fails when The system is fragile or the team is small. Automated chaos experiments is good for Continuous coverage, costs Needs guardrails and observability, and fails when Nobody reviews the results
Failure modes
- Standby credential gap. The switch fails because the backup provider key was never provisioned in production. Debug: run the switch with the primary disabled and read the exact error from the backup client.
- Capacity shortfall. The standby accepts traffic and immediately queues because its quota or replicas were sized for a fraction of load. Debug: compare inherited request rate with the standby's measured capacity during the drill.
- DNS or TTL delay. The application switched but traffic still reaches the failed endpoint until caches expire. Debug: measure the time from routing change to the first request arriving at the standby.
In production
Schedule drills on a calendar with an owner, and track mean time to switch as a metric rather than an anecdote. Keep the results in the runbook, including the steps that were wrong.
Common misconceptions
People think: Failover is proven by a successful configuration review.
Actually: Configuration describes intent; only an executed switch shows whether credentials, quotas and routing actually work.
People think: Testing failover once is enough.
Actually: Credentials rotate, quotas change and providers deprecate models, so the path decays unless it is exercised.
Interview angle
Keep detection and switching separate, name the credential and DNS paths, and verify standby capacity. Describe a bounded production drill and what gets written down. Having a backup provider configured, never run, is not a failover plan.
Sources
- Principles of Chaos Engineering — controlled experiments against steady-state hypotheses.
- AWS Builders' Library, Ensuring rollback safety during deployments — rehearsing reversal before it is needed.
- Google, SRE Book: Managing Incidents — practising response before real incidents.