Circuit breakers
Failing fast when a dependency is sick, and probing to recover.
01The intuition
The problem it solves
Your service calls a payment provider on every checkout. One afternoon the provider starts timing out: every call waits the full 2 seconds and then fails. Your servers each have 200 threads. Within a few seconds all 200 on every machine are stuck waiting on the provider. Requests that never needed the provider, like viewing the cart, now cannot get a thread either. The provider's outage has become your outage, and your retries are making the provider's recovery harder.
An electrical circuit breaker solves the same shape of problem in a house. When too much current flows, it trips and cuts the circuit, so the wiring does not melt. You reset it later, carefully. A software circuit breaker watches a dependency, trips when the dependency is failing, makes every call fail instantly instead of slowly, and periodically lets one call through to see whether the dependency has recovered.
Build it one piece at a time
Step 1: wrap the call and count. Every call to the provider goes through a wrapper that records success or failure. Keep a sliding window of the last 20 outcomes, or the last 10 seconds.
Step 2: three states. The breaker starts closed: calls pass through normally. If failures in the window cross a threshold, say 50 percent of at least 10 calls, it trips to open: every call returns an error immediately, in microseconds, without touching the network. After a cool-down, say 30 seconds, it moves to half-open: one trial call is allowed. If that succeeds, the breaker closes; if it fails, it opens again and the cool-down restarts.
| Time | Window outcomes | State | What callers experience |
|---|---|---|---|
| 12:00:00 | 20 successes | closed | normal, 150 ms |
| 12:00:08 | 6 of last 10 timed out | open | instant failure, 0.01 ms |
| 12:00:38 | cool-down over | half-open | one call tries the provider |
| 12:00:39 | trial failed | open | instant failure for 30 s more |
| 12:01:10 | trial succeeded | closed | normal again |
Step 3: decide what "fail instantly" means for the user. An open breaker on the payment provider should not show a blank page. Return a fallback: "payments are temporarily unavailable, your cart is saved", or queue the order for later processing, or use a cached answer where one is acceptable (a recommendation list from five minutes ago). The breaker protects your threads; the fallback protects the experience.
Step 4: one breaker per dependency, per server. Keep the state in each server's memory. A shared breaker would need a network call to check, which is what you are trying to avoid, and servers in different places may legitimately see the provider differently. A dependency with several endpoints gets a breaker per endpoint.
Step 5: pair it with a bulkhead. Even with a breaker, the first few seconds of an outage can consume all threads before it trips. Give each dependency its own small pool of threads or a concurrency limit, say 20 of the 200, so the provider can never take more than a tenth of a server.
Why it works, with the numbers
Without a breaker, during an outage each request to the provider holds a thread for the full timeout. With T threads, timeout t and arrival rate a of provider calls, threads are exhausted when
a × t ≥ T: at 150 calls/s and a 2 s timeout, 300 threads are needed, more than the 200 available
With the breaker open, each failed call costs microseconds, so a × t is near zero and all 200 threads serve other work. The provider receives one trial call every 30 seconds instead of 150 a second, so it can recover. The guarantee is that a failing dependency costs you a bounded number of threads and a bounded amount of its own capacity. The failure modes: a threshold so low that a few unlucky timeouts trip the breaker, a cool-down so long that recovery is slow, and a fallback that is itself broken.
The trade-off you must say out loud
You fail some calls that would have succeeded, in exchange for keeping your own service alive. Say the trip threshold, the cool-down, the half-open rule and the fallback for each dependency. Say the breaker is local, not shared. The follow-up is "how do you know it tripped", and the answer is a metric per breaker state and an alert on open.
Where it shows up in the 100 Google problems
- Slow and unreliable APIs: the breaker and the bulkhead are the core of the answer, with the fallback per call as the deep dive.
- API gateway: a breaker per upstream keeps one sick backend from stalling the gateway's threads.
- Notification system: when a push provider is down, the breaker keeps workers free for the other channels.
- E-commerce site: a tripped breaker on the recommendation service degrades the page instead of failing checkout.