Handling Replication Lag
What you'll learn: You'll understand three critical consistency problems that emerge when replicas fall behind the leader, and how they affect user experience.
The Core Problem
In asynchronous replication systems, followers can lag behind the leader by milliseconds to minutes. During this replication lag, clients may encounter confusing behaviors that violate their expectations about data consistency. Three classic problems arise:
Read-Your-Writes
You submit a form, then immediately refresh the page—but your changes are gone! This happens when your write goes to the leader, but your subsequent read hits a lagging follower that hasn't received the update yet.
Analogy: You mail a letter (write to leader), then immediately call the recipient to confirm (read from follower). They say "No letter yet!"—because postal delivery takes time.
Monotonic Reads
You refresh a page and see your friend's latest comment. Refresh again, and it disappears! This occurs when consecutive reads hit different replicas with different lag amounts, causing you to "go backward in time."
Analogy: Looking at two clocks in different rooms—the second clock is running slower, so time appears to move backward as you walk between them.
Consistent Prefix Reads
In a conversation, you see the answer before the question appears. This happens when related writes (like a question and its reply) are written to different partitions or replicas, and you observe them arriving out of causal order.
Analogy: Overhearing "I agree!" before hearing what the person agreed with—the causally later event arrived first.
Why It Matters
These aren't theoretical edge cases. Users experience them as bugs. A system with eventual consistency must either accept these anomalies or implement specific guarantees (like session consistency) to prevent them.
Key Takeaway: Replication lag creates three major consistency violations—read-your-writes, monotonic reads, and consistent prefix reads—each requiring specific handling strategies to maintain user trust.