Circuit Breaker
What it is
A failing service can waste a caller's time and resources. A circuit breaker stops calls once there have been enough failures in a row. Later, it lets one careful call through to check whether the service has recovered, instead of sending everything at once. This test call is sometimes called a probe. Catching the problem early, and checking it carefully before trusting the service again, keeps one outage from turning into a bigger one.
The core idea
There are three states:
- Closed: calls pass through as normal. Failures in a row are counted, and a success resets the count back to zero.
- Open: calls are rejected right away. After a wait, the breaker lets the next call through as a test instead of rejecting it.
- Half-open: only a small number of test calls are let through. Enough of them succeeding closes the breaker again. Even one of them failing sends it straight back to open, and the wait starts over.
Deciding whether to let a call through only needs the current time and the state already held. It must never pause or wait by itself. A failed test call starts a brand new wait, counted from that failure.
Example
With a failure limit of 3 and a wait of 60 ms: call 1 fails, call 2 fails, call 3 fails. Three in a row trips the breaker to open, so the next calls are rejected right away with no real attempt made. Once 60 ms have passed, the next call is let through as a test instead of being rejected. If that test call, and one more after it, both succeed, the breaker closes and traffic flows normally again. If a test call fails instead, the breaker goes straight back to open and the 60 ms wait starts over from that failure.
Why a careful test call, and not everything at once
Imagine the service only half recovered: it can handle a trickle of calls but would fall over again under full traffic. Sending every caller's calls through the moment the wait ends risks exactly that second collapse. Letting through just one or two test calls first limits the damage of a bad guess: if they succeed, it is reasonably safe to trust the service again, and if even one fails, the breaker can back off again before doing any real harm.
Where you meet this
- Martin Fowler's article describes the pattern: when many callers wait on a service that does not answer, they run out of threads and connections, and the failure spreads to every system that calls them. (CircuitBreaker)
- Resilience4j, a Java library, has the same three states. It opens on the failure rate over the last N calls or the last N seconds, where this challenge uses failures in a row. (Resilience4j CircuitBreaker)
- Hystrix, from Netflix, is a library built to stop failures from spreading between services. It is now in maintenance mode, and its page points new projects to Resilience4j. (Hystrix)
When to choose it: you call something over the network that can fail or hang, and a fast "not now" is better for your users than a long wait.
When not to: the failure comes from the request itself, like bad input. Those failures say nothing about the service, so they should not count. A breaker also adds little for a cheap local call that cannot hang.
A breaker decides whether to call at all. Retry with Backoff decides when to try again. The two work well together.
Common bugs
- Never opening, or never recovering.
- Keeping the failure count after a success instead of resetting it.
- Opening one failure too early or too late.
- Keeping the old wait start time after a failed test call, instead of starting a new wait from that failure.
- Letting concurrent callers change state without a lock.
Keep it fast
Keep a few counters and timestamps, not a growing history of past calls. Keep the lock short, and never hold it while making the real call to the service.
Learn more
Michael Nygard's Release It! and Martin Fowler's CircuitBreaker article explain the pattern. A circuit breaker decides whether to call a service at all. A retrier decides when to try again after a failure. The two pair naturally: the breaker guards the call, the retrier decides the timing around it.