A user edits their display name and hits save. The page reloads with the old name. They save again: old name. Thirty seconds later, without touching anything, the new name shows up in their activity feed. Twice.
The bug report reads like nonsense until you look at the infrastructure: writes go to the primary, reads come from a replica, and replication is asynchronous. The user wrote to one machine and read from another that hadn’t caught up. Nothing was down, nothing errored, every component did exactly what it was configured to do, and the system as a whole lied to the user about their own action.
Most of us file this under “replication lag” and move on. I think it deserves more attention, because that little bug is the CAP theorem showing up in a plain CRUD app. Not the conference-slide version; the real one. Knowing exactly what broke, and what the theorem does and doesn’t say about it, turns a slogan into a tool.
What CAP Actually Says
“Consistency, availability, partition tolerance: pick two” compresses a real theorem, conjectured by Eric Brewer in 2000 and proven by Seth Gilbert and Nancy Lynch in 2002, past the point of usefulness. Martin Kleppmann, author of Designing Data-Intensive Applications, spells out the precise definitions in his pointedly titled essay “Please stop calling databases CP or AP” (2015):
- Consistency in CAP means linearizability, a very specific and very strong guarantee. It has “nothing to do with the C in ACID.”
- Availability means every request received by a non-failing node gets a non-error response. Not “the system mostly works”: any node, every request. Plenty of systems sold as highly available don’t meet it.
- Partition tolerance, “terribly mis-named,” just means you talk over a network that can delay or drop messages. The internet and every datacenter have that property, so, in Kleppmann’s words, “you don’t really have any choice in this matter.”
That last point kills the “pick two” framing. Partitions aren’t optional, so the real content of the theorem is narrower: when a partition happens, you choose between linearizability and availability. Brewer conceded as much in his 2012 retrospective, “CAP Twelve Years Later”: “the ‘2 of 3’ formulation was always misleading because it tended to oversimplify the tensions among properties.”
Linearizability: The Guarantee You Think You Have
Kleppmann’s informal definition: “If operation B started after operation A successfully completed, then operation B must see the system in the same state as it was on completion of operation A, or a newer state.”
His illustration is the best one I know. Alice and Bob are in the same room during the 2014 World Cup final. Alice refreshes her phone, sees the final score and tells Bob. Bob refreshes his, the request lands on a lagging replica, and his phone says the game is still on. Bob’s query started after Alice’s had finished (he heard her say so, through a channel outside the database), so a linearizable system would have to show him a result at least as fresh as hers. It didn’t.
The display-name bug is the same story with Alice and Bob as one person. That special case has its own name, read-your-own-writes: the user’s second request started after their first one completed, so they expected to see its effect. The replica served them the past.
Why don’t databases just give everyone linearizability? Cost. It “requires a lot of coordination”: the system has to behave as though there’s a single copy of the data, no matter how many replicas and caches exist. Kleppmann’s most humbling example: “Even the CPU in your computer doesn’t provide linearizable access to your local RAM!” You need explicit memory barriers to get it. If linearizability is expensive inside one machine, expecting it for free across a replicated database was always optimistic.
Your Stack Is Eventually Consistent, NoSQL or Not
You don’t opt into eventual consistency by picking a trendy database. You get it the moment any copy of your data can be read while another copy is newer:
- Read replicas with asynchronous replication, the default in most MySQL and PostgreSQL setups.
- Caches. A Redis key with a 60-second TTL is a replica with a fixed staleness bound and no invalidation protocol.
- CDNs. Edge caches serve content seconds to minutes behind the origin.
- Search indexes. Documents indexed after commit, the exact scenario from the dual-write article, are a lagging replica of your database.
The fixes for read-your-own-writes are mechanical. Route a user’s reads to the primary for a short window after they write (a “recently wrote” flag in the session). Or record the primary’s log position after the write and only serve that user from a replica that has caught up. Or, simplest and most ignored, render the UI from the write response instead of fetching again. Each is about an hour of work. The expensive part is noticing which reads actually need the guarantee.
Which brings me to the one idea I’d keep from this whole article: consistency is per operation, not per database. A user reading their own profile edit needs read-your-own-writes. A user reading someone else’s profile can live with a few seconds of staleness and never notice. Same table, same query, different requirement.
“So Is My Database CP or AP?” Wrong Question
With the theorem in hand, the natural next move is to classify your stack: Postgres is CP, Cassandra is AP, done. Kleppmann’s essay exists to demolish that taxonomy, and his counterexamples are the databases you actually run:
- Single-leader replication, the standard relational setup. A client partitioned from the leader can’t write, so it isn’t CAP-available. Reads from asynchronous followers aren’t linearizable, so it isn’t CAP-consistent either. His verdict: “neither ‘CP’ nor ‘AP’, they are just ‘P’, whatever that means.”
- Dynamo-style stores such as Cassandra and Riak. CAP-available only in their weakest configuration (R=W=1). With quorum reads and writes, clients on the minority side of a partition can’t reach a quorum. And don’t count on quorums for linearizability either: sloppy quorums and read repair produce “tricky edge cases in which deleted data is resurrected.”
- ZooKeeper, everyone’s textbook “CP system,” serves reads from whichever node you’re connected to by default, which isn’t linearizable. You can get linearizable reads with
sync, and since 3.4.0 there’s a read-only mode that is CAP-available. Yet it “actually provides an excellent level of consistency,” stronger than many systems casually labeled AP.
His conclusion: “The fact that we haven’t been able to classify even one datastore as unambiguously ‘AP’ or ‘CP’ should be telling us something: those are simply not the right labels to describe systems.”
The model behind the theorem is narrow too: a single read-write register, one kind of fault, and nothing about latency. That last gap is what Daniel Abadi’s PACELC adds: even when there’s no partition (the “Else”), you trade Latency against Consistency. It matches daily life better: most asynchronous replication exists because cross-region round trips are slow, not because anyone planned for partitions. Kleppmann isn’t impressed by PACELC either (“I don’t find that any more enlightening than CAP”), and I see why: it’s still buckets. But the latency half is the trade-off you’ll actually make every week.
So is CAP worthless? No. As Kleppmann puts it, in 2000 “it was meant to start a discussion about trade-offs in distributed data systems, and it did that very well.” As a one-line vaccine against believing you can have perfect consistency and perfect availability over an unreliable network, it still works. What fails is the classification scheme people built on top of it. The Jepsen project’s map of consistency models shows the real landscape: not two buckets but a dozen-plus models (linearizable, sequential, causal, read-your-writes, monotonic reads, eventual), each a different contract with a different price.
When NOT to Pay for Strong Consistency
Strong consistency is the safe default for correctness, which is exactly why you should know where it’s the wrong purchase:
- Cross-region, user-facing reads. Linearizable reads across continents put cross-continent round trips on your hot path. Nearly every global product serves regional replicas and accepts bounded staleness, because the latency win is worth more than the freshness.
- Data nobody re-reads right away. Analytics events, logs, metrics: consumers aggregate them later, and small delays and reorderings wash out.
- Naturally convergent state. View counters, like counts, presence dots. Approximately right now beats exactly right later, and reconciling them is trivial.
- Reads with no causal link to the reader. The home feed, someone else’s profile, a product listing. Nobody can notice staleness in data they’ve never seen the fresh version of.
The mirror list, where you must have it, is short and non-negotiable: anything the user just wrote and will immediately look at; anything that gates a decision (inventory before a sale, balance before a withdrawal); anything where two clients seeing different values triggers different side effects.
The Audit
- Draw the copies. For one critical entity (a user, an order), list every place a copy lives: primary, replicas, caches, CDN, search index, client state. Every arrow between them has a staleness bound. Write it down. “Unknown” is a finding.
- Find the read-your-own-writes paths. For every flow where a user writes and then immediately reads: does that read hit a replica or a cache? If it does, you have the bug from the opening, whether or not anyone has reported it yet.
- Classify your reads, not your database. Per endpoint: linearizable, read-your-writes, or eventual is fine. You’ll find the strong bucket is smaller than you feared, and currently unenforced.
- Check how your framework routes reads. If it sends every read to replicas indiscriminately, pin reads to the primary after writes for the flows in the second bucket. A few hours of work, and a whole class of bugs disappears.
- Retire “CP/AP” from your design docs. Name the actual guarantee per operation. Precision is free here and saves entire meetings.
Next in the series: observability, because none of this is debuggable if you can’t tell which replica served the stale read.
References
- Martin Kleppmann — Please stop calling databases CP or AP, 2015
- Eric Brewer — CAP Twelve Years Later: How the “Rules” Have Changed, IEEE Computer, 2012 (PDF)
- Seth Gilbert & Nancy Lynch — Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services, ACM SIGACT News, 2002 (PDF)
- Daniel Abadi — Problems with CAP, and Yahoo’s little known NoSQL system, 2010
- Jepsen — Consistency Models
- Martin Kleppmann — A Critique of the CAP Theorem, arXiv, 2015