Avoiding Hotspots in Sharding
You'll learn how to prevent some shards from becoming overloaded while others sit idle.
In sharded databases, a hotspot occurs when one shard receives disproportionately more traffic than others. This defeats the purpose of sharding—you've distributed your data, but not your load. The system becomes as slow as its busiest shard.
Why Hotspots Happen
The most common culprit is monotonically increasing keys like auto-incrementing user IDs or timestamps. If you shard by user_id using range-based sharding (e.g., users 1-1000 go to Shard A, 1001-2000 to Shard B), all new users always land on the latest shard. That shard gets hammered with writes while older shards sit quiet.
Similarly, certain patterns create "celebrity" problems—think of a viral tweet. If you shard tweets by tweet_id but one tweet goes viral, the shard holding that tweet becomes a bottleneck.
Prevention Strategies
Introduce randomness into your shard key distribution:
- Use hash-based sharding instead of ranges to scatter sequential IDs uniformly
- Add random prefixes or suffixes to keys before hashing
- Choose composite shard keys that naturally distribute load (e.g.,
user_id + timestamp_bucket) - Avoid time-based keys alone; combine them with user or entity identifiers
Analogy: Imagine assigning students to lunch tables by birth order (hotspot: everyone born this year crowds one table) versus by a hash of their full name (even distribution).
Monitor shard metrics continuously—CPU, query rates, storage—to detect emerging hotspots early.
Key Takeaway: Avoid hotspots by choosing shard keys that distribute data evenly and unpredictably, steering clear of monotonically increasing values that concentrate new activity on one shard.