Why it matters
When a provider or downstream service degrades, continuing to call it consumes connection slots and threads, so the caller fails slowly instead of quickly and the dependency cannot recover. A breaker converts that into a bounded wait: after enough failures, calls return immediately, freeing capacity and giving the dependency room. The design questions are where the state lives, what counts as a failure, and how the probe is limited.
Mental model
It is a fuse in a circuit: it melts before the wiring does, and you replace it once. The analogy breaks because a software breaker resets itself, and the reset is the risky moment when every caller tries again at once.
How it works
- 1Wrap one dependency, not a whole request, so a breaker on retrieval does not disable generation.
- 2Define what counts as a failure: connection errors, timeouts and specific status codes, but not user errors.
- 3Set the window and threshold from the dependency's normal error rate, so routine noise does not open the circuit.
- 4Fail fast while open with a typed error that the fallback ladder can handle.
- 5Allow a limited number of probe requests in half-open, and close only after consecutive successes.
- 6Export the state as a gauge and log transitions with the reason, so an incident timeline shows when the circuit changed.
Step 1 of 6
Wrap one dependency, not a whole request, so a breaker on retrieval does not disable generation.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Wrap one dependency, not a whole request, so a breaker on retrieval does not disable gene… 2. Define what counts as a failure: connection errors, timeouts and specific status codes, b… 3. Set the window and threshold from the dependency's normal error rate, so routine noise do… 4. Fail fast while open with a typed error that the fallback ladder can handle. 5. Allow a limited number of probe requests in half-open, and close only after consecutive s… 6. Export the state as a gauge and log transitions with the reason, so an incident timeline …
- Wrap one dependency, not a whole request, so a breaker on retrieval does not disable generation.
- Define what counts as a failure: connection errors, timeouts and specific status codes, but not user errors.
- Set the window and threshold from the dependency's normal error rate, so routine noise does not open the circuit.
- Fail fast while open with a typed error that the fallback ladder can handle.
- Allow a limited number of probe requests in half-open, and close only after consecutive successes.
- Export the state as a gauge and log transitions with the reason, so an incident timeline shows when the circuit changed.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Per-process breaker | Simple, no coordination | Each instance opens separately | Traffic is uneven across instances |
| Shared breaker state | Consistent fleet behaviour | A shared store on the failure path | The store itself is unavailable |
| Low threshold | Fast protection | Trips on normal noise | The dependency has a naturally high error rate |
| Long open interval | Gives the dependency time to recover | Extended degradation for users | The dependency recovers quickly |
Good for
Simple, no coordination
Cost
Each instance opens separately
Fails when
Traffic is uneven across instances
- Traffic is uneven across instances
- The store itself is unavailable
- The dependency has a naturally high error rate
- The dependency recovers quickly
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Per-process breaker is good for Simple, no coordination, costs Each instance opens separately, and fails when Traffic is uneven across instances. Shared breaker state is good for Consistent fleet behaviour, costs A shared store on the failure path, and fails when The store itself is unavailable. Low threshold is good for Fast protection, costs Trips on normal noise, and fails when The dependency has a naturally high error rate. Long open interval is good for Gives the dependency time to recover, costs Extended degradation for users, and fails when The dependency recovers quickly
Failure modes
- Breaker without a timeout. Calls hang until the client timeout and the failure counter never increments. Debug: assert every wrapped call has a deadline shorter than the request budget.
- Half-open flood. All waiting requests probe at once and knock the dependency over again. Debug: count probe requests per second after a transition to half-open.
- Wrong failure classification. Validation errors from a healthy dependency open the circuit and disable a working path. Debug: chart breaker state against error codes and confirm the counts exclude client errors.
In production
Alert on time spent open, not just on transitions, and review the failure classification whenever a provider changes its error format. Rehearse recovery with a dependency stub that fails and then succeeds slowly.
Common misconceptions
People think: A circuit breaker improves availability by itself.
Actually: It shortens failure; availability comes from what the caller does while the circuit is open.
People think: Retries and breakers are alternatives.
Actually: Retries handle transient blips, breakers handle sustained failure; using only retries turns a degradation into a storm.
Interview angle
Walk the three states and the transition conditions, then the scope of the breaker and what the caller returns while it is open. A strong answer mentions timeouts and the limited probe. Weak answers describe a retry counter with no recovery policy.
Sources
- Martin Fowler, CircuitBreaker — the closed, open and half-open states.
- Microsoft, Circuit Breaker pattern — thresholds, recovery probes and interaction with retries.