Circuit breakers, retries and bulkheads

Cascading failure dynamics and thread pool starvation

In a call tree, failure travels upward. Consider an orchestrator with $100$ worker threads that calls several downstream services and holds a thread for each call in flight.

Suppose one non-critical dependency, Recommendations, slows from $20\text{ ms}$ to $4{,}000\text{ ms}$. By Little's law (chapter 1), the threads busy with it equal the call rate times the call duration. So at $4\text{ s}$ per call, just $100 / 4 = 25$ calls per second to Recommendations hold all $100$ threads. At $20\text{ ms}$, the same threads could serve $5{,}000$ calls per second.

Once every thread is waiting on Recommendations, checkout and payment requests have no thread to run on either. Fowler puts it this way: with many callers on an unresponsive supplier "you can run out of critical resources leading to cascading failures across multiple systems."

The circuit breaker (three states)

Fowler: "You wrap a protected function call in a circuit breaker object, which monitors for failures."

             failure rate ≥ threshold
   ┌────────┐ ───────────────────────> ┌────────┐
   │ CLOSED │                          │  OPEN  │  calls rejected
   └────────┘ <──────┐        ┌─────── └────────┘  without a network call
                     │        │  trial calls      │
       trial calls   │        │  fail too often   │ wait duration
       mostly succeed│        │                   │ elapses
                     │   ┌───────────┐ <──────────┘
                     └── │ HALF-OPEN │
                         └───────────┘

Exact behavior varies by library. The numbers below are resilience4j's documented defaults:

1. Closed: calls pass through, and outcomes are recorded in a sliding window. In the default count-based window, that is the last $100$ calls. No failure rate is computed until a minimum number of calls ($100$ by default) is recorded. When the failure rate reaches the threshold ($50\%$ by default), the breaker opens.
2. Open: calls are rejected at once with a CallNotPermittedException, so the caller can serve a fallback without waiting on the network. The breaker stays open for a configured wait ($60\text{ s}$ by default).
3. Half-open: a limited number of trial calls ($10$ by default) are let through. If their failure rate is still at or above the threshold, the breaker opens again; otherwise it closes. Fowler's original description uses a single trial call.

The AWS Builders' Library adds a caution: circuit breakers "introduce modal behavior into systems that can be difficult to test".

Retry storms and backoff with jitter

When calls fail because the service is overloaded, "retries can make the overload worse instead of better" (AWS Builders' Library). If every call fails, each layer that retries multiplies the load:

So $8{,}000\text{ RPS}$ with $\text{MaxRetries} = 3$ becomes $32{,}000\text{ RPS}$ at the failing service. Retries at several layers multiply too. AWS's example is a five-layer stack where each layer tries three times: the database sees $3^5 = 243$ times the load, "making it unlikely to ever recover". Retry only at one layer, and cap retries, for example with a token bucket that allows retries only while tokens remain.

Full jitter (Marc Brooker's AWS post, chapter 7) spreads the retries that remain:

Here $B$ is the base delay, $M$ the cap and $i$ the retry number. With the example values $B = 100\text{ ms}$ and $M = 10\text{ s}$, the ceiling doubles 100, 200, 400 … ms until it reaches 10 s at the eighth retry. Clients that all failed at the same moment then retry at different moments instead of in lockstep.

Bulkheads

The pattern is named after the sectioned partitions of a ship's hull: "if the hull of a ship is compromised, only the damaged section fills with water" (Azure Architecture Center). A consumer gives each dependency its own pool, such as a connection pool, thread pool or semaphore limit. Then "if a service begins to fail, it only affects the connection pool assigned for that service." In the example above, a separate pool for Recommendations would leave checkout's threads free.

The price is less efficient use of resources. Azure suggests combining bulkheads with retries, circuit breakers and throttling.