Hinted Handoff and Read Repair
You'll learn how Cassandra maintains data consistency even when nodes temporarily fail.
The Problem: Temporary Node Unavailability
In a distributed wide-column store like Cassandra, nodes occasionally go down for maintenance, network issues, or crashes. But clients still need to write data. How do you handle writes destined for an unavailable node without losing them?
Hinted Handoff: Temporary Stand-Ins
When a coordinator node tries to write to a replica but finds it unavailable, it doesn't fail the write. Instead, it stores a hint (a note about the write) on another healthy node. Think of it like leaving a package with a neighbor when someone isn't home.
Once the unavailable node comes back online, the node holding the hint detects this (via the gossip protocol) and "hands off" the stored writes to the now-recovered replica. The data eventually reaches its intended destination.
Tradeoff: Hints are stored for a limited time (typically a few hours). If a node stays down too long, hints expire and other repair mechanisms take over.
Read Repair: Fixing Inconsistencies on Reads
Even with hinted handoff, replicas can drift out of sync. Read repair fixes this during read operations.
When a coordinator reads from multiple replicas (based on your consistency level), it compares the returned data timestamps. If replicas disagree, the coordinator identifies the most recent version and silently pushes updates to the outdated replicas in the background.
This ensures that frequently-read data converges toward consistency over time—another form of eventual consistency at work.
Key Takeaway: Hinted handoff temporarily stores writes for downed nodes (like leaving a note with a neighbor), while read repair detects and fixes stale data during reads, both helping Cassandra achieve eventual consistency despite node failures.