When a dependency starts failing in a distributed system, the instinct to keep trying can make everything worse. Circuit breakers solve this by detecting failures and temporarily stopping requests to unhealthy services, giving them time to recover while protecting the rest of your infrastructure from cascading collapse.

The Cascading Failure Problem

A single slow or failing service can bring down an entire system. When Service A calls Service B and B starts timing out, A’s threads pile up waiting for responses that never arrive. Connection pools exhaust. Memory fills with queued requests. Eventually A stops responding to its own callers, and the failure ripples upstream through every service that depends on it.

The default behavior—retry until success—accelerates the problem. Each retry adds load to an already struggling service. Timeouts burn resources on both sides. What started as a database connection issue in one microservice becomes a company-wide outage.

How Circuit Breakers Work

Circuit breakers sit between your service and its dependencies, monitoring request outcomes and enforcing three states: closed, open, and half-open.

In the closed state, requests flow normally. The breaker tracks failures—timeouts, errors, or response codes above a threshold. When failures exceed a configured rate or count within a time window, the breaker trips.

In the open state, the breaker immediately fails requests without attempting to contact the dependency. This is the core protection: rather than wasting threads and connections on a service that’s clearly down, you fail fast and return an error or fallback response. The breaker remains open for a timeout period, giving the downstream service time to recover without additional load.

After the timeout expires, the breaker enters half-open state. It allows a limited number of test requests through. If they succeed, the breaker resets to closed and normal traffic resumes. If they fail, it reopens and waits longer before trying again.

Configuration and Thresholds

Effective circuit breakers require tuning. Set the failure threshold too low and transient network blips trigger false opens. Set it too high and the breaker trips after significant damage is done.

Most implementations use a sliding window—tracking the last N requests or the last T seconds—rather than a simple counter. A threshold like “50% failure rate over 10 seconds with at least 20 requests” prevents premature tripping from a single error while responding quickly to sustained failures.

Timeout values matter equally. Open too briefly and you hit the dependency before it recovers, resetting the clock. Open too long and you delay recovery unnecessarily. Typical production values range from 5 to 60 seconds depending on the dependency’s recovery characteristics.

Fallback Strategies

An open circuit breaker stops the cascading failure, but your service still needs to respond. Fallback strategies depend on the operation. For non-critical features, you can degrade gracefully—skip recommendations, hide social features, or return cached data. For critical paths, you might return a default value, queue the request for later processing, or fail with a clear error message.

The key is designing systems where not every dependency is critical for every request. Circuit breakers work best when coupled with feature flagging, caching layers, and asynchronous processing that can tolerate temporary unavailability.

Observability and Testing

Circuit breaker state changes are high-signal events. When a breaker opens, you want alerts, dashboards, and logs that capture the failure pattern. Metrics should track not just state transitions but also rejected request counts, half-open probe results, and time spent in each state.

Testing circuit breakers requires fault injection. Chaos engineering tools can simulate downstream failures, verify that breakers trip at the right thresholds, and confirm that services degrade gracefully rather than falling over. Load testing should include scenarios where dependencies fail to ensure your breaker configuration protects the system under real traffic volume.

Circuit breakers are a pattern, not a panacea. They prevent cascading failures and buy time for recovery, but they don’t fix the underlying issue. When a breaker opens, something downstream is broken—and that still needs to be debugged and resolved.