Fixed vs Dynamic Partitioning
What you'll learn: How choosing between a fixed number of partitions versus letting partitions split and merge impacts scalability, complexity, and operational burden.
The Core Choice
When designing a partitioned system, you face a fundamental decision: fixed partitioning pre-allocates a specific number of partitions upfront (say, 256 partitions), while dynamic partitioning allows partitions to split when they grow too large or merge when they shrink too small.
Think of it like office space planning. Fixed partitioning is like building 100 cubicles on day one—simple to manage, but you're stuck with that number. Dynamic partitioning is like movable walls that adjust as teams grow or shrink—more flexible, but requires active management.
Fixed Partitioning
With fixed partitioning, you create all partitions at system initialization. Each partition handles a subset of your key space. When adding nodes, you rebalance existing partitions across more machines, but the partition count stays constant.
Pros:
- Simple mental model and predictable behavior
- No runtime partition management overhead
- Secondary indexes remain stable
Cons:
- Must estimate capacity upfront (over-provision = waste, under-provision = performance degradation)
- Cannot adapt to uneven data growth patterns
- Rebalancing shuffles entire partitions between nodes
Dynamic Partitioning
Dynamic partitioning starts small and splits partitions automatically when they exceed size thresholds, distributing the new partitions across nodes as needed. This is what you saw in "Dynamic Range Splitting."
Pros:
- Adapts naturally to actual data distribution
- Efficient use of resources—partitions exist only when needed
- Can handle unpredictable growth patterns
Cons:
- More complex implementation and debugging
- Split/merge operations create temporary load spikes
- Requires careful threshold tuning to avoid thrashing
Key Takeaway: Fixed partitioning offers operational simplicity at the cost of flexibility, while dynamic partitioning provides automatic adaptation but adds runtime complexity and coordination overhead.