Timeouts, Retries, and Backoff: The Resilience Primitives Nobody Configures

Here is the zero value that has taken down more services than any exploit:

var client http.Client
// client.Timeout == 0 — which net/http documents as "no timeout"

Go’s http.Client, used without configuration, waits forever. So does Python’s requests unless you pass timeout=. So do plenty of database drivers. The default behavior of most networking code, across most ecosystems, is unbounded patience — and unbounded patience is how one slow dependency converts into a fleet-wide outage.

The mechanics are worth feeling concretely. Every in-flight request holds resources: a goroutine, a connection from a bounded pool, memory, maybe a database transaction. When a downstream service stops answering — not erroring, just hanging — those resources don’t get released. They accumulate. Marc Brooker, Senior Principal Engineer at AWS, describes the endgame in the Amazon Builders’ Library article “Timeouts, retries, and backoff with jitter”: “When a number of requests hold on to resources for a long time, the server can run out of those resources. These resources can include memory, threads, connections, ephemeral ports, or anything else that is limited.”

Your service didn’t crash because the recommendation engine went down. It crashed because it waited politely, with every thread it had, for a service that was never going to answer. The three primitives that prevent this — timeouts, retries, backoff with jitter — are conceptually trivial and configured wrong almost everywhere. This article is about the values: how Amazon actually chooses them, and where the intuitive choices amplify failures instead of containing them.

Timeouts: A Number, Not a Feeling

Amazon’s stated practice, per Brooker, is “to set a timeout on any remote call, and generally on any call across processes even on the same box” — both a connection timeout and a request timeout. The hard part isn’t setting one; it’s choosing the number. Too high, and it doesn’t protect you. Too low, and — Brooker lists the two risks precisely — you get “increased traffic on the backend and increased latency because too many requests are retried,” and in the worst case a small latency increase “leading to a complete outage, because all requests start being retried.”

The method Amazon uses in-region is refreshingly mechanical, not vibes-based:

Decide your acceptable rate of false timeouts — requests that would have succeeded, killed by the timeout. Brooker’s example: 0.1%.
Set the timeout at the corresponding latency percentile of the downstream service — for 0.1%, that’s the p99.9.

That’s it. The timeout becomes a derived quantity of measured latency, revisited when the latency distribution changes — not a folk number like “30 seconds” inherited from a tutorial.

Brooker documents the pitfalls with a war story: one Amazon system set a 20-millisecond timeout that worked fine — except immediately after deployments, when requests mysteriously timed out. The cause: the timer included TLS connection establishment, which took longer than 20ms, so every fresh server’s first requests died. The team’s eventual fix was establishing connections at process startup, before taking traffic. The general lessons: know exactly what your timeout covers (DNS? TLS handshake? just the read?), pad tight-latency services where p99.9 sits close to p50, and factor in real network latency for internet-facing clients.

In Go, the layered version looks like this:

client := &http.Client{
    Timeout: 800 * time.Millisecond, // end-to-end cap: DNS+connect+TLS+read
    Transport: &http.Transport{
        DialContext: (&net.Dialer{
            Timeout: 200 * time.Millisecond, // connection establishment only
        }).DialContext,
        TLSHandshakeTimeout:   200 * time.Millisecond,
        ResponseHeaderTimeout: 500 * time.Millisecond,
    },
}

// And per-request deadlines that flow through the whole call tree:
ctx, cancel := context.WithTimeout(ctx, 800*time.Millisecond)
defer cancel()
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)

The context deadline is the mechanism Brooker’s article alludes to when noting Go provides “more robust timeout mechanisms” than raw socket options: it propagates through every downstream call that respects ctx, giving you a deadline for the operation rather than a pile of disconnected per-socket timers.

Retries: A Medicine With a Maximum Dose

Retries work because, as Brooker notes, distributed systems mostly suffer partial failures (some percentage of requests fail) and transient failures (failures for a short window) — and the same request, tried again, usually succeeds. But his framing carries a warning the industry keeps ignoring: “Retries are ‘selfish.’ In other words, when a client retries, it spends more of the server’s time to get a higher chance of success.” His metaphor: “a powerful medicine — useful in the right dose, but can cause significant damage when used too much.”

The overdose scenario has a number attached. Consider a five-layer call stack ending in a database, with three retries at each layer. When the database starts failing under load, the retries multiply through the layers: 3 × 3 × 3 × 3 × 3. Brooker: “the load on the database will increase 243x, making it unlikely to ever recover.” Every layer independently doing the “resilient” thing produces a system that clubs its own database to death. Amazon’s practice: “retry at a single point in the stack.”

Two more rules that cost nothing and prevent real incidents:

Don’t retry client errors. A 400 or 404 will fail identically on attempt two; retrying it is pure waste. HTTP’s client/server error split encodes retryability — though Brooker honestly notes eventual consistency “blurs this line,” since a 404 for a just-created resource may succeed moments later.
Don’t retry non-idempotent operations blind. Amazon’s position: “APIs with side effects aren’t safe to retry unless they provide idempotency” — which is why EC2’s RunInstances accepts an explicit client token, and why the previous article on idempotency is a prerequisite for everything in this one. A timeout doesn’t mean the side effect didn’t happen. It means you don’t know.

Backoff and Jitter: The Half Everyone Skips

Retrying immediately against a struggling server is how a blip becomes an outage. The standard fix is capped exponential backoff — wait 100ms, 200ms, 400ms… up to a ceiling. Necessary, and by itself, insufficient — for a reason that’s easy to miss: correlation.

When a dependency hiccups, it fails many clients at the same moment. If they all back off by the same deterministic schedule, they all come back at the same moment too — a synchronized wave, arriving again and again. Brooker: “If all the failed calls back off to the same time, they cause contention or overload again when they are retried.”

The fix is jitter — randomizing each client’s wait. Brooker’s earlier analysis on the AWS Architecture Blog, “Exponential Backoff and Jitter” (2015), simulated the variants and found the counterintuitive winner is full jitter: pick uniformly at random between zero and the exponential cap, rather than adding a little noise to a fixed wait. In Go:

func backoff(attempt int) time.Duration {
    const base, cap = 100 * time.Millisecond, 10 * time.Second
    exp := min(cap, base*(1<<attempt)) // capped exponential
    return time.Duration(rand.Int64N(int64(exp))) // full jitter: [0, exp)
}

Stripe’s engineering team reached the same design for their client libraries — exponential backoff plus jitter — explicitly to avoid the thundering herd of synchronized clients hammering a recovering server.

And jitter’s usefulness extends beyond retries, in a way most teams never apply: Brooker recommends jittering all periodic work — cron jobs, refresh timers, batch schedules — because fleets of clients firing “at the first few seconds of a minute, or the first few seconds after midnight” create invisible load spikes. On Amazon EBS and Lambda, spreading clients’ periodic work let them serve “the same amount of work with less server capacity.” One subtlety: for scheduled jobs Amazon uses jitter that’s consistent per host rather than fully random, so when something does overload, it overloads in a reproducible pattern a human can debug.

The Budget: What Bounds the Whole Thing

Even with backoff and jitter, retries still multiply traffic during a real outage — capped exponential backoff eventually has “all of the clients retrying constantly at the capped rate.” Something must bound the total retry volume.

The reader who’s been following this series might object here: isn’t this what circuit breakers are for? Interestingly, Brooker is lukewarm on them — a rare public disagreement with popular resilience wisdom worth taking seriously: “circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery.” A breaker is a mode switch, and systems with modes fail in mode-transition ways. Amazon’s preferred alternative is a retry token bucket: retries spend tokens, tokens refill slowly, and when the bucket is empty, requests fail fast instead of retrying. The effect is proportional throttling — full retry behavior in good times, bounded retry rate in bad times, no cliff. AWS built exactly this into its SDKs in 2016, so every SDK call already carries the behavior.

Google’s SRE book describes the same idea from the server-protection angle in its chapter on addressing cascading failures: per-client retry budgets, so no client converts an outage into a self-sustaining retry storm. Circuit breakers aren’t wrong — the next articles in this series cover where they shine — but the token bucket is the simpler default, and it’s telling which one Amazon reached for.

When NOT to Add All This

The primitives are cheap, but they’re not free everywhere. Skip or simplify when:

The call is in-process or to localhost. A context deadline is still wise; three-tier retry machinery around a local cache is ceremony.
A human is watching and will retry anyway. For interactive requests, one fast retry beats a multi-second backoff schedule — the user is the retry loop, and they have their own timeout.
You’d be the fourth retrying layer. If your SDK retries (AWS SDKs do) and your service mesh retries and your gateway retries, adding application retries recreates Brooker’s 243x multiplication in miniature. Find where retries already happen, and retry at one layer.
The operation isn’t idempotent yet. Fix that first; retries against a non-idempotent mutation are a duplication engine.

The Audit

Five checks, one afternoon, against your own codebase:

Hunt zero values. Every http.Client, DB pool, and RPC stub: does it have an explicit timeout? The unconfigured ones are your outage inventory.
Justify each number. For each timeout, can you name the downstream percentile it derives from? “It was in the example code” is the wrong answer; p99.9 + acceptable-false-timeout-rate is the right shape.
Find every retry loop and check for jitter. Backoff without randomness is synchronized retry waves waiting for their first real outage.
Count your retrying layers. Trace one user request through SDKs, meshes, gateways, and application code. If more than one layer retries, you’ve found your multiplication factor.
Bound the total. Add a token bucket or retry budget so the retry rate during an outage is a number you chose, not an emergent property.

Timeouts decide when you give up, retries decide whether to try again, backoff with jitter decides when — and a budget decides how much of it you can afford. Four small decisions, all with well-documented right answers. The next article climbs one level up: what “consistency” actually means when your data lives in more than one place.