Your client sends a payment request. Three seconds pass. Timeout.

Quick question: did the charge happen?

Think about what the timeout actually tells you. Maybe the request never reached the server. Maybe it arrived, the charge was processed, and the response got lost on the way back. Maybe the server crashed halfway through. From the client’s side, these three scenarios are indistinguishable — same timeout, same error object, same nothing.

Now the client faces a bad menu. Don’t retry, and a real customer’s payment may have silently vanished. Retry, and you may charge them twice. Both options are wrong, and no amount of clever client-side logic can fix it — because the problem lives in the API’s design, not the client’s.

There is exactly one property that dissolves this dilemma. It’s called idempotency: designing operations so that performing them many times has the same effect as performing them once. Once an endpoint is idempotent, the client’s menu collapses to a single safe item: retry until you get a definitive answer. This article covers how the companies that process money at scale actually implement it — and how to build the same guarantee in your own backend.

The Ambiguity Is Fundamental, Not Fixable

The three-way ambiguity above isn’t an implementation detail you can engineer away. Stripe engineer Brandur Leach, in the company’s engineering post “Designing robust and predictable APIs with idempotency” (2017), enumerates the failure cases for any call between two nodes:

His point: “the success of the operation is ambiguous from the perspective of the client, and it doesn’t know whether retrying the operation is safe.” And this applies to any distributed system — which, as Leach notes, means “as few as two computers connecting via a network that are passing each other messages.”

AWS reaches the same conclusion from the provider side. In the Amazon Builders’ Library article “Making retries safe with idempotent APIs” (January 2021), Principal Engineer Malcolm Featonby documents the strategies Amazon uses so that clients can retry mutating operations — launching instances, provisioning resources — without duplicating side effects. When both Stripe and Amazon independently converge on the same mechanism for their most money-sensitive operations, that mechanism is worth understanding precisely.

If you read the previous article on the dual-write problem, this is the same lesson from the other side of the wire: there, ambiguous failure corrupted your own system’s consistency; here, it corrupts the contract between you and your callers.

Two Kinds of Idempotency

Natural idempotency: operations that carry their own safety

Some operations are idempotent by construction. Setting status = "shipped" on order 42 can run five times; the result is identical. HTTP’s semantics already encode this: per RFC 7231, PUT and DELETE are defined as idempotent verbs — PUT means “create or replace this resource with this exact content,” which is repetition-safe by definition.

Leach’s example is a DNS API: a PUT to create a CNAME record can be invoked any number of times. “If the server receives a call that it realizes is a duplicate because the domain already exists, it simply ignores the request and responds with a successful status code.”

The design lesson: prefer absolute state over relative change. SET balance = 100 is naturally idempotent; ADD 10 TO balance never can be. When you control the operation’s shape, shape it absolutely.

Idempotency keys: manufacturing safety for operations that create

But “charge this customer $20” can’t be phrased as absolute state — it inherently creates something new each time. For these operations, the client generates a unique idempotency key per logical operation and sends it with the request. The server uses the key to detect repeats.

Stripe’s API reference documents the exact mechanics, and the details are more instructive than the concept:

That second detail is the one most homegrown implementations miss: Stripe caches failures too. The replay guarantee is about giving the client a consistent view of what happened, not about pretending everything succeeded.

Building It: The Idempotency Table

Here’s the core of a server-side implementation in Go with PostgreSQL. The schema:

CREATE TABLE idempotency_keys (
    key          TEXT PRIMARY KEY,
    request_hash TEXT NOT NULL,
    status_code  INT,
    response     JSONB,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

And the handler logic:

func (s *Server) WithIdempotency(
    ctx context.Context, key, reqHash string,
    op func(ctx context.Context) (int, []byte, error),
) (int, []byte, error) {
    // Claim the key. ON CONFLICT DO NOTHING makes concurrent
    // duplicates race safely: exactly one caller wins the insert.
    res, err := s.db.ExecContext(ctx,
        `INSERT INTO idempotency_keys (key, request_hash)
         VALUES ($1, $2) ON CONFLICT (key) DO NOTHING`, key, reqHash)
    if err != nil {
        return 0, nil, err
    }

    if rows, _ := res.RowsAffected(); rows == 0 {
        // Key exists: either a finished request (replay its response)
        // or one still in flight (tell the client to wait).
        var prevHash string
        var code sql.NullInt64
        var body []byte
        err := s.db.QueryRowContext(ctx,
            `SELECT request_hash, status_code, response
             FROM idempotency_keys WHERE key = $1`, key,
        ).Scan(&prevHash, &code, &body)
        if err != nil {
            return 0, nil, err
        }
        if prevHash != reqHash {
            return http.StatusUnprocessableEntity,
                []byte(`{"error":"idempotency key reused with different request"}`), nil
        }
        if !code.Valid {
            return http.StatusConflict,
                []byte(`{"error":"original request still in progress"}`), nil
        }
        return int(code.Int64), body, nil
    }

    // We own the key: run the real operation and record its outcome —
    // success or failure alike, exactly as Stripe does.
    code, body, opErr := op(ctx)
    _, _ = s.db.ExecContext(ctx,
        `UPDATE idempotency_keys SET status_code = $2, response = $3
         WHERE key = $1`, key, code, body)
    return code, body, opErr
}

Three decisions in this code deserve emphasis. The ON CONFLICT DO NOTHING insert is the concurrency guard — two simultaneous requests with the same key can’t both execute the operation, because only one insert wins. The request_hash comparison catches key reuse with different payloads, mirroring Stripe’s behavior. And the in-flight case (status_code IS NULL) returns a conflict instead of blocking — the client’s retry loop will come back.

Add a cleanup job that deletes rows older than your retention window (Stripe’s 24 hours is a sane default), and document that window in your API reference — it defines how long a client’s retry remains safe.

Webhooks: Where You’re on the Other Side

Everything above assumed you’re the server. With webhooks, you’re the consumer — and the sender has the same ambiguity problem you had. When a provider (Stripe, WhatsApp, GitHub) doesn’t receive a timely 2xx from your endpoint, it retries. That’s not a bug; it’s the sender correctly refusing to lose events. The consequence is a hard rule: any webhook handler that isn’t idempotent will eventually process duplicates.

In my experience this is not theoretical. Two systems I maintain receive webhooks from messaging providers, and duplicate deliveries arrive routinely — after our endpoint was slow to respond, after provider-side incidents, after network flaps. The handler pattern that survives: extract the provider’s event ID, attempt an INSERT ... ON CONFLICT DO NOTHING into a processed_events table keyed on it, and skip processing when the insert affects zero rows. Same table trick as above, one column smaller.

The same pattern extends to client-side sync queues. ScrumBoard, my offline-first project management app, queues user operations in the browser while offline and replays them on reconnect — and replay means possible duplicates by design. Every queued operation carries a client-generated ID that the server deduplicates on, which is what makes the sync loop safe to run any number of times. That design is a full article later in this series.

The Objection: “Isn’t This Just Extra Complexity?”

A fair pushback: you’ve now added a table, a race-safe claim protocol, a hash check, and a cleanup job — for requests that succeed on the first try 99.9% of the time. Is it worth it?

Sometimes it isn’t, and the honest engineering answer is to say so. If the operation is naturally idempotent (state-setting PUTs), the table adds nothing — HTTP semantics already protect you, and per Stripe’s docs, sending idempotency keys on GET and DELETE “has no effect” because “these requests are idempotent by definition.” If a duplicate costs little (an extra log line, a re-sent notification that providers dedupe anyway), the machinery may cost more than the failure. And idempotency keys protect single operations — they don’t give you cross-operation transactions, and pretending they do creates subtler bugs than the ones they fix.

The decision hinges on one question: what does one duplicate cost? A duplicate charge, order, or shipment justifies the full table. A duplicate cache refresh doesn’t. Most systems contain both — the skill is not defaulting to either extreme.

There’s also a client-side half of this contract that idempotency alone doesn’t cover: how you retry. Leach’s post insists retries use exponential backoff with jitter, so a fleet of failing clients doesn’t synchronize into a thundering herd against an already-struggling server. That protocol — timeouts, backoff, jitter, retry budgets — is the subject of an upcoming article in this series.

Make Your API Retry-Safe This Week

  1. Inventory your mutating endpoints. For each POST, ask: what happens if this exact request executes twice? If the answer involves money, inventory, or emails, it needs a key.
  2. Reshape what you can. Any operation expressible as absolute state (PUT /orders/42/status) gets idempotency for free. Do this before building infrastructure.
  3. Add the key mechanism to what remains. One table, the claim-then-execute pattern above, response caching for failures included, documented retention window.
  4. Harden every webhook handler. Provider event ID + unique constraint + skip-on-conflict. Assume duplicates; the provider assumes you do.
  5. Test it the honest way. Fire the same request at your endpoint twice concurrently — not sequentially — and verify one side effect. The concurrent case is the one that finds the missing ON CONFLICT.

Idempotency is the first pattern in this series that pays off on day one: it requires no new infrastructure, no broker, no consensus protocol — just a table and the discipline to route mutations through it. The next article covers what happens when even retries have rules: delivery semantics, and why “exactly-once” is a promise nobody can actually make.

References