Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 28 of 38

Failover testing

0 of 38 complete

Why it matters

Failover involves more than a conditional in the client: credentials must exist for the backup, quotas must be raised, DNS or routing must point elsewhere, and the fallback model must produce acceptable answers. Each is a separate failure point that only appears when the primary is actually gone. Testing converts the unknown into a measured switch time and a list of things that did not work.

Mental model

It is a fire drill: the point is not the fire, it is discovering that the stairwell door is locked. The analogy breaks because a drill can be scheduled at a convenient time, while the real failure arrives at peak traffic with the on-call engineer asleep.

How it works

Mechanism, step by stepInteractive
  1. 1Enumerate the failure scenarios: provider outage, region loss, quota exhaustion, model deprecation, credential expiry.
  2. 2For each, write the expected detection signal, the switch mechanism and the target recovery time.
  3. 3Run the scenario in staging first, then in production with a bounded blast radius such as one tenant or one route.
  4. 4Time detection and switch separately, and record where the delay came from.
  5. 5Check the standby's capacity and quality under the inherited load, not just that it responds.
  6. 6Update the runbook with what actually happened, including the commands that were needed and missing.

Step 1 of 6

Enumerate the failure scenarios: provider outage, region loss, quota exhaustion, model deprecation, credential expiry.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Enumerate the failure scenarios: provider outage, region loss, quota exhaustion, model de… 2. For each, write the expected detection signal, the switch mechanism and the target recove… 3. Run the scenario in staging first, then in production with a bounded blast radius such as… 4. Time detection and switch separately, and record where the delay came from. 5. Check the standby's capacity and quality under the inherited load, not just that it respo… 6. Update the runbook with what actually happened, including the commands that were needed a…

  1. Enumerate the failure scenarios: provider outage, region loss, quota exhaustion, model deprecation, credential expiry.
  2. For each, write the expected detection signal, the switch mechanism and the target recovery time.
  3. Run the scenario in staging first, then in production with a bounded blast radius such as one tenant or one route.
  4. Time detection and switch separately, and record where the delay came from.
  5. Check the standby's capacity and quality under the inherited load, not just that it responds.
  6. Update the runbook with what actually happened, including the commands that were needed and missing.

Trade-offs

ChoiceGood forCostFails when
Staging-only drillsNo customer riskMisses production credentials, DNS and quotasThe failure involves infrastructure configuration
Production with small blast radiusRealistic and boundedSome users affectedThe route cannot be isolated
Full production failoverProves end-to-end capacityLarge risk and coordination costThe system is fragile or the team is small
Automated chaos experimentsContinuous coverageNeeds guardrails and observabilityNobody reviews the results
Trade-off mapInteractive

Good for

No customer risk

Cost

Misses production credentials, DNS and quotas

Fails when

The failure involves infrastructure configuration

  1. The failure involves infrastructure configuration
  2. The route cannot be isolated
  3. The system is fragile or the team is small
  4. Nobody reviews the results

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Staging-only drills is good for No customer risk, costs Misses production credentials, DNS and quotas, and fails when The failure involves infrastructure configuration. Production with small blast radius is good for Realistic and bounded, costs Some users affected, and fails when The route cannot be isolated. Full production failover is good for Proves end-to-end capacity, costs Large risk and coordination cost, and fails when The system is fragile or the team is small. Automated chaos experiments is good for Continuous coverage, costs Needs guardrails and observability, and fails when Nobody reviews the results

Failure modes

  • Standby credential gap. The switch fails because the backup provider key was never provisioned in production. Debug: run the switch with the primary disabled and read the exact error from the backup client.
  • Capacity shortfall. The standby accepts traffic and immediately queues because its quota or replicas were sized for a fraction of load. Debug: compare inherited request rate with the standby's measured capacity during the drill.
  • DNS or TTL delay. The application switched but traffic still reaches the failed endpoint until caches expire. Debug: measure the time from routing change to the first request arriving at the standby.

In production

Schedule drills on a calendar with an owner, and track mean time to switch as a metric rather than an anecdote. Keep the results in the runbook, including the steps that were wrong.

Common misconceptions

People think: Failover is proven by a successful configuration review.

Actually: Configuration describes intent; only an executed switch shows whether credentials, quotas and routing actually work.

People think: Testing failover once is enough.

Actually: Credentials rotate, quotas change and providers deprecate models, so the path decays unless it is exercised.

Interview angle

Keep detection and switching separate, name the credential and DNS paths, and verify standby capacity. Describe a bounded production drill and what gets written down. Having a backup provider configured, never run, is not a failover plan.

Sources