Rebalancing Partitions
What you'll learn: How distributed systems redistribute data when nodes are added or removed without bringing everything to a halt.
When you partition data across nodes, your system's capacity isn't static—nodes crash, new servers get added, or traffic patterns change. Rebalancing is the process of redistributing data across the cluster when the number of partitions or nodes changes.
The Challenge
Imagine a library that's split books across 5 rooms based on author name ranges (A-E, F-J, etc.). When you add a 6th room, you need to reshuffle books—but patrons are still borrowing! Rebalancing must happen while the system stays operational.
Fixed Number of Partitions
One strategy is to create many more partitions than nodes from the start. For example, with 10 nodes you might create 1000 partitions. Each node manages ~100 partitions initially.
When you add an 11th node, you simply move whole partitions (not individual records) from existing nodes to the new one. Each node now handles ~91 partitions. No recalculation of keys needed—partitions are stable units.
Pros: Simple, predictable
Cons: Must choose partition count upfront; too few limits future scalability, too many wastes resources
Dynamic Splitting
Partitions start large and split when they exceed a size threshold, like cells dividing. A partition with 100GB might split into two 50GB partitions when crossing a limit.
When you add nodes, the system can move these dynamically-sized partitions to balance load. HBase and MongoDB use this approach.
Pros: Adapts to data growth automatically
Cons: More complex coordination; splitting can temporarily impact performance
The Balancing Act
Both strategies share a goal: minimize data movement while achieving even distribution. Moving less data means faster rebalancing and less strain on the network.
Key Takeaway: Rebalancing redistributes partitions across nodes when capacity changes; fixed partitions offer simplicity through pre-allocated units, while dynamic splitting adapts partitions to actual data size—both keep the system available during the shuffle.