Consistent Hashing for Minimal Resharding
You'll learn: How consistent hashing dramatically reduces the amount of data that needs to move when you add or remove shards from your database.
The Problem with Traditional Hashing
Imagine you have 5 shards and use simple hash-based sharding: shard = hash(key) % 5. This works fine until you need to scale to 6 shards. Now the formula becomes hash(key) % 6, and suddenly almost every key maps to a different shard. You'd need to move roughly 80% of your data!
How Consistent Hashing Solves This
Consistent hashing arranges shards on a virtual "ring" (imagine a clock face from 0 to 360 degrees). Each key is also hashed to a position on this ring, and it belongs to the first shard encountered moving clockwise.
The magic happens during resharding: When you add a new shard to the ring, it only "claims" keys from its immediate neighbors. When you remove a shard, only its keys need to move to the next shard on the ring.
The Math Behind the Win
With traditional hashing and N shards, adding one shard requires moving approximately (N/(N+1)) of all data. With 5→6 shards, that's 83% of data.
With consistent hashing, adding a shard only requires moving approximately 1/(N+1) of data. That's just 17% — a massive improvement!
This makes resharding operations feasible in production without lengthy maintenance windows or massive data transfer operations.
Key Takeaway: Consistent hashing minimizes data movement during resharding by localizing the impact of shard changes to only the affected portions of the hash ring, rather than remapping everything.