Open your codebase and search for a function that does these two things, in this order: write something to the database, then tell another system about it. Publish a message to a queue. Invalidate a cache. Update a search index. Call a webhook.

Found one? Almost certainly. It’s one of the most natural shapes in backend code:

func (s *OrderService) PlaceOrder(ctx context.Context, o Order) error {
    if err := s.db.InsertOrder(ctx, o); err != nil {
        return err
    }
    // The gap. Right here.
    if err := s.queue.Publish(ctx, OrderPlaced{ID: o.ID}); err != nil {
        return err // ...and now what?
    }
    return nil
}

Now ask the uncomfortable question: what happens if the process crashes between the two calls? Or the network drops exactly there? Or the deploy restarts the pod at that instant?

The order exists in the database. The event was never published. The shipping service never hears about it. No error fired anywhere — the insert succeeded, and the process died before it could fail at anything. Your system is now inconsistent, and it will stay inconsistent, silently, until a customer asks where their package is.

This is the dual-write problem, and once you learn to see it, you’ll find it everywhere — including in systems you shipped years ago. Understanding it precisely is the difference between architectures that degrade gracefully and architectures that lie to you.

Where the Problem Was Named

The clearest formulation comes from Martin Kleppmann, in his 2015 Craft Conference talk, published as “Using logs to build a solid data infrastructure (or: why dual writes are a bad idea)”. Kleppmann’s definition is disarmingly simple: dual writes happen whenever “it’s your application code’s responsibility to update data in all the right places” — database, cache, search index, message broker — as separate operations.

His verdict, after walking through the failure modes: it’s “a really bad idea, because it has some fundamental problems.”

AWS formalized the same issue in its Prescriptive Guidance on cloud design patterns: “A dual write operation occurs when an application writes to two different systems… A failure in one of these operations might result in inconsistent data.” Their canonical example is a flight-booking service that must persist the booking and notify a payment service — and they document, diagram by diagram, how every naive ordering of those two operations fails.

The problem is well-documented, has been for a decade, and is still shipped into production every day — because the code that contains it looks completely innocent.

The Two Failure Modes

Dual writes fail in two distinct ways, and it’s worth separating them, because developers usually defend against one while remaining fully exposed to the other.

Partial failure: the crash in the gap

Kleppmann illustrates this with a denormalized counter: a messaging app inserts a message into a user’s inbox, then increments their unread-message counter. If the process dies between the two writes — database restart, crash, unplugged network cable — “your database is inconsistent: the message has been added to the inbox, but the counter hasn’t been updated… it will forever remain inconsistent.”

The key word is forever. This is not a transient error that resolves on retry, because there is no error. Nothing failed loudly. The first write committed; the second never started. No exception handler ran, because the process that would run it was gone.

And the reverse ordering is worse. If you publish the event first and the database write then fails and rolls back, downstream services are now acting on something that never happened. AWS’s flight-booking walkthrough shows this exact case: the transaction rolls back, “but the event notification might still be sent, causing the payment service to process the payment” — for a flight that was never booked.

Race conditions: the interleaving you can’t see

The second failure mode doesn’t require anything to crash. Kleppmann’s example: two clients concurrently update key X across two datastores — a database and a search index. Client one writes X=A to both; client two writes X=B to both. Requests interleave so the database applies A then B, but the index receives B then A.

Final state: database says B, search index says A. Every individual write succeeded. As Kleppmann puts it: “the two datastores are inconsistent with each other, and they will permanently remain inconsistent until sometime later someone comes and overwrites X again.”

His most quotable warning is about detection: “you probably won’t even notice that your database and your search indexes have gone out of sync, because no errors occurred. You’ll probably only realize six months later, while you’re doing something completely different.”

Why try/catch Solves Nothing

The instinctive fix is error handling. Catch the publish failure, and… what, exactly?

Retry the publish? Reasonable — unless the failure was a crash, in which case there’s no process left to retry. A retry loop only helps with failures your process survives. The gap failure kills the process itself.

Roll back the database write? You can’t. It’s committed. You could issue a compensating delete — but the compensation is itself a write that can fail, and now you’ve recursed into the same problem one level down.

Log it for manual repair? The log write can also fail, and even when it works, you’ve converted a correctness guarantee into an ops chore that someone must notice, triage, and replay by hand.

The mechanism to internalize is this: two systems, two commits, no shared transaction. The database has ACID atomicity inside its own boundary. The moment your unit of work spans the database and Kafka, or the database and Redis, atomicity evaporates — there is no coordinator making those two commits stand or fall together. No arrangement of if err != nil can conjure one into existence, because error handling runs inside your process, and the failure modes that matter kill or bypass your process.

Kleppmann anticipated the classic objection — “transactions solved this decades ago” — and answered it: atomicity works “in databases that support it,” but the moment the second write targets a different system, “you lose the ability to tie the writes together into a single transaction.” Distributed transactions via two-phase commit exist, but as we’ll see later in this series, 2PC trades away availability and throughput so aggressively that mainstream message brokers don’t even offer it.

The Honest Counterpoint: Sometimes It’s Fine

Before the solutions — is this always worth fixing? No, and pretending otherwise is its own architectural failure.

If the second write is a cache invalidation with a short TTL, the inconsistency window is bounded by the TTL. Self-healing by design. If the downstream consumer runs a periodic full reconciliation — nightly re-index, hourly sync — dual writes ripped in the meantime get repaired on schedule. And if the event is genuinely fire-and-forget (an analytics ping, a non-critical notification), the cost of occasionally losing one may be far below the cost of the machinery that would guarantee it.

The dual-write problem is a real defect when the second write carries state that must not diverge — money, orders, inventory, anything a downstream system treats as truth. Engineering judgment is knowing which of your dual writes are in that category. Most systems have both kinds.

The Real Solutions (a Preview)

Every correct solution shares one idea: stop making two writes, and make one write that produces two effects.

The transactional outbox. Write the business row and the event into the same database, in the same local transaction — the event goes into an outbox table. One commit, atomic by the database’s own ACID guarantees. A separate relay process reads the outbox and publishes to the broker, retrying until it succeeds. AWS’s guidance documents this as the standard resolution, with one crucial caveat: the relay delivers at least once, so “we recommend that you make the consuming service idempotent by tracking the processed messages.” We’ll build a full outbox implementation in Go later in this series, and idempotency is the subject of the next article.

Change data capture (CDC). Instead of an outbox table, treat the database’s own replication log as the event stream — tools like Debezium, or DynamoDB Streams in AWS’s example, capture committed changes and forward them. The database’s commit is the single write; everything else derives from it.

Log-first architecture. Kleppmann’s more radical proposal: don’t write to the database first at all. Append every change to a log (Kafka), and let the database, cache, and search index all consume the log in order. Ordering conflicts disappear because every consumer sees the same sequence. This is a bigger architectural commitment — few teams start here, but it’s where the reasoning leads.

Find Yours Today

A 20-minute audit that’s worth running on any codebase you own:

  1. Grep for the shape. Search for publish/send/invalidate/index calls (Publish(, SendMessage, Del(, Index() and check what happens in the same function before them. A DB write above and a network call below, with no outbox between them, is a hit.
  2. Classify each hit. Self-healing (TTL, reconciliation job) or divergence-critical (money, orders, state machines)? Only the second category is a defect.
  3. Check the ordering. Event-before-commit is strictly worse than commit-before-event: it announces things that may never become true. Fix those first.
  4. Don’t reach for a distributed transaction. The fix is the outbox or CDC — one atomic local write, then asynchronous, retried delivery.
  5. Write the finding down. Even if you fix nothing this week, an inventory of your dual writes turns an invisible bug class into a managed risk.

The dual-write problem is the entry ticket to distributed systems thinking: the moment you truly accept that two systems cannot be updated atomically from application code, the rest of the patterns in this series — outboxes, idempotent consumers, delivery semantics — stop looking like ceremony and start looking inevitable.

References