Delivery Semantics: At-Most-Once, At-Least-Once, and the Exactly-Once Myth

In 2015, engineer Tyler Treat published an essay whose title has become a distributed-systems proverb: “You Cannot Have Exactly-Once Delivery”. Two years later, Confluent — the company founded by Kafka’s creators — published a post titled, almost defiantly: “Exactly-once Semantics is Possible: Here’s How Apache Kafka Does it”.

Both are written by people who deeply understand messaging systems. Both are still cited today. They appear to flatly contradict each other.

They don’t — and the resolution of this apparent contradiction is one of the most practically useful ideas in backend engineering. It determines how you configure every queue you’ll ever use, and it explains a class of production incidents that otherwise look like vendor bugs. The short version: one essay is about delivery, the other is about processing, and conflating those two words is where systems quietly lose or duplicate data.

The Three Semantics

Any messaging system — Kafka, SQS, RabbitMQ, or two services passing HTTP calls — offers one of three guarantees about a message, as laid out in Confluent’s documentation on delivery semantics:

At-most-once. The sender fires and forgets. The message arrives once or not at all — never twice. Acknowledge before processing, and a crash mid-processing loses the message forever. As Treat puts it bluntly: “Customer transaction? Sorry, looks like you’re not getting your order.”

At-least-once. The sender waits for an acknowledgment and re-sends until it gets one. Acknowledge after processing, and a crash between processing and acking means the sender redelivers — the message arrives again. Nothing is lost; things are duplicated.

Exactly-once. Every message arrives once, no losses, no duplicates. The one everybody wants — and, at the transport level, the one nobody can have.

Notice the structural symmetry: the only difference between at-most-once and at-least-once is when you acknowledge relative to when you process. That one-line ordering decision in your consumer is the entire semantic. Most teams make it implicitly, by copying an example from the docs, without realizing they’ve chosen which failure mode they prefer.

Why Exactly-Once Delivery Is Impossible

The impossibility isn’t an engineering limitation awaiting a clever fix. It’s a provable property of unreliable communication, and the canonical illustration is the Two Generals Problem: two armies must attack simultaneously to win, coordinating by messenger through enemy territory where any messenger can be captured. General A sends “attack at dawn.” Did it arrive? A needs a confirmation. But B’s confirmation can also be lost — so B needs a confirmation of the confirmation. Every message needs an ack, and every ack is itself a message. There is no finite number of messages after which both generals are certain. The problem has no solution — not “no known solution,” but provably none.

Treat maps this directly onto messaging: “we try to guarantee the delivery of a message by waiting for an acknowledgement that it was received, but all sorts of things can go wrong. Did the message get dropped? Did the ack get dropped? Did the receiver crash? Are they just slow?” His conclusion is worth quoting exactly, because it’s the sentence the industry keeps relearning: “FLP and the Two Generals Problem are not design complexities, they are impossibility results.”

The sender that hasn’t received an ack faces the same fork every time: re-send (risking a duplicate — at-least-once) or don’t (risking a loss — at-most-once). There is no third option at the transport layer. Even RabbitMQ’s own reliability documentation concedes the point, as Treat highlights: after a connection failure, producers should retransmit unconfirmed messages, “there is a possibility of message duplication here… Therefore consumer applications will need to perform deduplication or handle incoming messages in an idempotent manner.”

So What Is Kafka Claiming?

Here’s where the two essays stop contradicting each other. Kafka 0.11 (2017) shipped two features under the “exactly-once semantics” banner, documented in Confluent’s announcement:

The idempotent producer. Each producer gets an ID and attaches sequence numbers to messages. When a retry causes the same message to be sent twice, the broker detects the duplicate sequence number and writes it to the log only once. Read that carefully: the duplicate delivery still happens — the broker deduplicates on arrival. This is at-least-once delivery plus deduplication, mechanized inside the broker.

Transactions. A producer can write to several partitions (including the consumer-offsets partition) atomically, so “consume → process → produce” pipelines commit or abort as a unit. Failures cause re-processing, but aborted results are never visible to downstream readers configured with read_committed.

Neither feature makes messages traverse the network exactly once — retries and redeliveries still occur constantly. What Kafka guarantees is that the observable effect on the log is as if each message were handled once. Treat named this move years before, without the marketing gloss: “The way we achieve exactly-once delivery in practice is by faking it. Either the messages themselves should be idempotent… or we remove the need for idempotency through deduplication.”

The precise vocabulary, which Treat’s 2017 follow-up insists on: delivery is a transport semantic and cannot be exactly-once; processing is an application semantic and can be. Kafka’s achievement is real and engineering-impressive — it’s exactly-once processing within Kafka’s boundary. The moment your consumer touches anything outside that boundary — your database, an email API, a payment provider — you’re back outside the guarantee, and it’s your job again.

This is a live debate, not a settled score. Commenters on Treat’s post — and engineers at companies selling stream processors — argue the distinction is pedantic, that “exactly-once has always meant at-least-once plus dedup” and customers successfully rely on it. That’s fair as far as it goes. The danger isn’t in the feature; it’s in the engineer who reads “exactly-once” on a datasheet and concludes their consumer code no longer needs to handle duplicates. It does.

The Practical Recipe

Strip away the theory and a simple, universal design falls out — the same one used by every serious system:

Choose at-least-once delivery, and make processing idempotent.

In Go, against any at-least-once source (SQS, RabbitMQ, Kafka without transactions), the consumer shape is:

func (c *Consumer) Handle(ctx context.Context, msg Message) error {
    // Dedup on the message's stable ID inside the same transaction
    // as the side effect — this is what makes redelivery harmless.
    tx, err := c.db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    res, err := tx.ExecContext(ctx,
        `INSERT INTO processed_messages (id) VALUES ($1)
         ON CONFLICT (id) DO NOTHING`, msg.ID)
    if err != nil {
        return err
    }
    if n, _ := res.RowsAffected(); n == 0 {
        return tx.Commit() // duplicate: ack it, skip the work
    }

    if err := applyBusinessEffect(ctx, tx, msg); err != nil {
        return err // no ack → broker redelivers → we retry
    }
    return tx.Commit() // effect + dedup record commit atomically
}

The load-bearing detail: the dedup insert and the business effect share one database transaction. Written separately, a crash between them recreates the dual-write problem covered earlier in this series — and this handler pattern is the mirror image of the idempotency-key table from the previous article. The three first articles of this series are one design, seen from three angles.

And for what can’t be processed: dead letter queues. At-least-once means retrying forever is a real failure mode — a malformed message (“poison pill”) redelivers infinitely, burning consumer capacity. Every mainstream broker supports routing a message to a DLQ after N failed attempts. Configure it, alert on non-empty DLQs, and build a replay path. A DLQ nobody monitors is just at-most-once with extra steps.

When the Other Semantics Are Right

At-least-once + idempotency is the default, not a law.

At-most-once is correct when staleness beats duplication. Metrics samples, presence pings, live cursor positions: a lost reading is replaced by the next one in seconds, while machinery to guarantee its delivery adds latency to something whose value expires immediately. Fire-and-forget is the honest design.

Kafka transactions are correct when the pipeline lives inside Kafka. For stream-processing topologies (Kafka in, Kafka out), the transactional API gives you exactly-once processing without hand-rolling dedup tables — that’s precisely its design domain. The cost is coordination overhead and configuration complexity, which is why it’s opt-in.

When NOT to build any of this: if your “queue” is a single process feeding itself (a Go channel, an in-memory job list), there is no network and no ambiguity — adding distributed-systems ceremony to a monolith’s internals is architecture cosplay. The semantics matter exactly when a network boundary sits between producer and consumer.

Audit Your Consumers Tomorrow

Find every consumer’s ack point. Ack-then-process is at-most-once; process-then-ack is at-least-once. Verify each one matches what the business actually tolerates — loss or duplication.
For every at-least-once consumer, find the dedup. If there isn’t one, you have a duplicate-processing bug with a fuse of unknown length. The table above is an afternoon of work.
Check dedup and effect share a transaction. Side-by-side writes without atomicity is the dual-write problem wearing a queue costume.
Confirm every queue has a DLQ with an alert. Ask: “what happens to a message that fails 100 times?” If the answer is “it retries forever,” fix that this week.
Reread any vendor’s ‘exactly-once’ claim and identify the boundary it applies to. Inside the boundary: their guarantee. Outside: your dedup table.

The three semantics aren’t a menu where one option is simply best — they’re an honest accounting of what an unreliable network permits. The systems that behave well in production are the ones whose designers knew exactly which guarantee they were getting, and built the missing half themselves.