Let’s reconstruct an outage. The recommendations service, the “customers also bought” widget, starts responding slowly. Not failing. Slowly. Checkout doesn’t depend on recommendations. Checkout goes down anyway.

The autopsy: product pages call recommendations, and on that path nobody set a sane timeout. Each call holds a thread and a connection for seconds instead of milliseconds. Traffic keeps arriving, held resources pile up, and the shared connection pool runs dry. Now every feature that needs a connection, checkout included, waits in line behind ghosts. Nothing crashed. The service filled up with waiting.

That’s a cascading failure: one component’s degradation spreading through shared resources and synchronous calls until unrelated features die. Timeouts shrink the damage per request, but under enough traffic even short timeouts, multiplied by thousands of requests, drain a pool. The next line of defense is structural: stop calling the sick service, and stop letting features share their fate. Those two moves are the circuit breaker and the bulkhead. Michael Nygard popularized both in Release It! (2007, 2nd edition 2018), and Martin Fowler’s 2014 bliki entry spread the breaker everywhere.

The Circuit Breaker: Failing Fast on Purpose

The metaphor is the panel in your house: when a circuit overloads, the breaker trips before the wiring burns. Fowler’s summary of the software version: “You wrap a protected function call in a circuit breaker object, which monitors for failures. Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all.”

The state machine has three positions:

In Go the well-worn implementation is sony/gobreaker. Its v2 is generic, and configuring it forces exactly the right questions:

import "github.com/sony/gobreaker/v2"

cb := gobreaker.NewCircuitBreaker[[]Recommendation](gobreaker.Settings{
    Name:         "recommendations",
    Interval:     time.Minute,      // counting window...
    BucketPeriod: 10 * time.Second, // ...rolled in 10s buckets (omit it and the window is fixed)
    Timeout:      30 * time.Second, // open → half-open cooldown
    ReadyToTrip: func(c gobreaker.Counts) bool {
        return c.Requests >= 10 &&
            float64(c.TotalFailures)/float64(c.Requests) > 0.5
    },
    // A caller that hung up is not evidence that the dependency is sick.
    IsExcluded: func(err error) bool { return errors.Is(err, context.Canceled) },
    OnStateChange: func(name string, from, to gobreaker.State) {
        slog.Warn("breaker state change", "breaker", name, "from", from.String(), "to", to.String())
    },
})

recs, err := cb.Execute(func() ([]Recommendation, error) {
    return fetchRecsWithBulkhead(ctx, productID)
})
if err != nil {
    recs = nil // fallback: render the page without the widget
}

Two details people miss. Without Interval, gobreaker never clears its counts while closed, so “50% of recent requests” quietly becomes “50% since the last state change”. And not every error should count: Fowler makes the same point, some failures “should reflect normal failures and be dealt with as part of regular logic.” A 404 or a validation error says nothing about the dependency’s health.

The last line of the snippet is the part teams skip and regret: a breaker without a fallback just turns slow failures into fast ones. Fast is already a win, because your resources survive, but the user still sees a broken feature. The fallback is a product decision dressed as code: an empty widget, the last known good response, default content. For recommendations, rendering nothing is fine. For payment authorization there’s no silent fallback. Your options are a fast, honest error, or what Fowler suggests, putting the authorization on a queue to handle later, which means shipping before you’re sure you’ve been paid. That’s a business call, not a config value.

Bulkheads: Sharing Is How Failures Travel

The breaker protects you from a dependency you already know is sick. The bulkhead protects you from the sharing that let the sickness spread. The name comes from ship design: the hull is split into watertight compartments, so a breach floods one of them, not the ship.

In the outage above, recommendations and checkout drew from the same connection pool. Give them separate compartments and recommendations can drown its own without touching checkout’s:

func newClient(timeout time.Duration, maxConns int) *http.Client {
    // Clone the default transport so you keep its proxy, dial and TLS timeouts.
    // A bare &http.Transport{} silently drops all of them.
    t := http.DefaultTransport.(*http.Transport).Clone()
    t.MaxConnsPerHost = maxConns
    return &http.Client{Timeout: timeout, Transport: t}
}

var (
    checkoutClient = newClient(500*time.Millisecond, 100) // critical path: its own generous pool
    recsClient     = newClient(300*time.Millisecond, 10)  // nice-to-have: small, capped compartment
    recsSem        = make(chan struct{}, 10)               // hard cap on concurrent recommendation calls
)

func fetchRecsWithBulkhead(ctx context.Context, id string) ([]Recommendation, error) {
    select {
    case recsSem <- struct{}{}:
        defer func() { <-recsSem }()
    default:
        return nil, ErrBulkheadFull // shed immediately instead of queueing
    }
    return fetchRecommendations(ctx, recsClient, id)
}

Ten connections of damage is now the maximum recommendations can cause, however sick it gets. The semaphore generalizes to anything scarce: worker goroutines, database connections, memory-heavy operations. Separate deployments and container resource limits are the same pattern at a coarser grain. Match the unit of isolation to the unit of failure you can tolerate.

The hidden prerequisite: bulkheads force you to rank your features. Giving checkout 100 connections and recommendations 10 is a statement about what the business can afford to lose. Google made that ranking a first-class field of its RPC system: every request carries a criticality (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE), and overloaded backends reject the lower classes first. Teams that never rank explicitly have ranked implicitly: everything shares everything, so every feature is exactly as critical as the flakiest one in the process.

Load Shedding: Reject Early and Cheaply

Breakers and bulkheads handle a sick dependency. When you are the one overloaded, the tool is load shedding: turning away excess work at the front door. The SRE book’s chapter on handling overload sets the bar: “a backend task provisioned to serve a certain traffic rate should continue to serve traffic at that rate without any significant impact on latency, regardless of how much excess traffic is thrown at the task.” The default: branch in the semaphore above is shedding in miniature: ErrBulkheadFull costs almost nothing, while queueing the request would cost memory, latency and eventually the process.

A fast 503 is kinder than a slow 200 that becomes a timeout. Clients with retry budgets and backoff handle the 503 gracefully. Accepting work you can’t finish degrades every request instead of a fraction, and the timeouts arrive anyway, later and more expensively.

One caveat from the same chapter: rejecting isn’t free. For a cheap request, “it’s almost equally expensive to reject a request that requires a simple RAM lookup … as it is to accept and run that request,” and a backend can end up spending most of its CPU just saying no. That’s why Google pairs server-side shedding with client-side throttling: clients that see a lot of rejections start failing requests locally, before they ever reach the network. Make sure your 503 path touches no database and allocates as little as possible.

Amazon Doesn’t Love Circuit Breakers

Resilience content usually presents the breaker as settled best practice. It isn’t, and the strongest dissent comes from inside the house. Marc Brooker’s Builders’ Library article on timeouts, retries and backoff is direct: “circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery.” Amazon’s answer to retry amplification is a token bucket for retries: everything retries while there are tokens, then retries continue at a fixed rate. AWS added that to its SDKs in 2016.

In a later post on his blog, Brooker sharpens the argument into one line: “Circuit breakers are designed to turn partial failures into complete failures.” Picture a sharded database where only the shard holding keys A–H is overloaded. A client-side breaker sees the failure rate climb, trips, and now requests for S–Z fail too, even though that shard is perfectly healthy. In sharded and cell-based architectures, a breaker can defeat the very isolation those architectures were built for. His follow-up compares a variant: a breaker that only stops retries while first attempts keep flowing.

Where I land: use a breaker where the failure mode is binary by nature, a dependency that’s either working or hard-down, with a meaningful fallback on the open path. That’s the recommendations widget, not your sharded storage layer. For gradual degradation, reach for the modeless tools first: deadlines, retry budgets, bulkheads, shedding. And give every breaker a metric and a log line on state changes, as Fowler also recommends. An open breaker nobody notices is a feature silently amputated.

When NOT to Use These Patterns

The Resilience Audit, Round Two

  1. Map shared fates. For each feature, list the pools, threads and clients it shares with others. Every shared resource is a path failure can travel. Decide which features may share a compartment.
  2. Rank the paths. Critical (checkout, auth) vs. degradable (recommendations, badges). Write it down: it’s the input for every bulkhead size and every fallback.
  3. Give degradable paths a cap, a fallback and, only if it’s binary, a breaker. Confirm each fallback is a product-approved experience, not just nil.
  4. Add front-door shedding. A concurrency cap with instant 503s beats an unbounded queue. Check that your 503 path touches no database.
  5. Instrument the modes. Breaker state changes, bulkhead rejections and shed counts are all metrics, all alertable. Silent degradation is the failure mode of the failure-handling machinery itself.
  6. Test the transitions. Kill the dependency in staging and watch: does the breaker trip, does the fallback render, does it recover when the dependency returns? Untested resilience machinery is decoration.

Cascading failure is the characteristic death of distributed systems: every component works and the connections between them kill you. Next in the series: drawing lines inside the codebase, with hexagonal architecture in a real Go service.

References