Consistent Hashing for Sharding
What you'll learn: How consistent hashing minimizes chaos when your sharded database needs to grow or shrink.
The Resharding Problem
When you use simple hash-based sharding (like hash(user_id) % num_shards), adding or removing a shard is painful. Change the number of shards, and almost every key maps to a different shard—triggering massive data migration. If you go from 4 to 5 shards, roughly 80% of your data needs to move!
How Consistent Hashing Helps
Consistent hashing places both shards and data keys on a circular hash ring (imagine a clock face). Each key hashes to a position on the ring and gets assigned to the next shard encountered clockwise.
The magic: When you add or remove a shard, only data near that position on the ring needs to move. Instead of affecting 80% of keys, consistent hashing typically redistributes only 1/n of the data (where n is the number of shards).
The Ring Analogy
Think of a circular running track with water stations (shards). Runners (data) start at their hash position and run clockwise to their nearest water station. Add a new station? Only runners between the new station and the previous one switch—everyone else keeps their assignment.
Virtual Nodes Enhancement
To avoid uneven distribution, each physical shard gets multiple positions on the ring (called "virtual nodes"). This smooths out data distribution and makes adding/removing shards more gradual.
Practical Impact
Consistent hashing makes dynamic sharding feasible. You can scale your database horizontally without planning massive downtime windows for data migration—critical for growing systems that can't afford service interruptions.
Key Takeaway: Consistent hashing transforms sharding from a one-time architectural decision into a flexible scaling strategy by limiting data movement to only the affected portion of the ring when shards change.