Directory-Based Partitioning
You'll learn: How to use a lookup service to map keys to shards, enabling flexible partition assignment that can be changed without reorganizing all your data.
The Core Idea
In previous partitioning strategies like range-based or hash-based partitioning, the algorithm itself determines which shard holds a given key. Directory-based partitioning takes a different approach: it maintains a separate lookup table (the "directory") that explicitly stores which partition holds each key or key range.
Think of it like a phone book for your data. Instead of calculating "Person A lives at address hash(A) % num_servers," you simply look up "Where does Person A live?" in your directory and get back "Server 3."
How It Works
The directory service acts as a central registry that answers the question: "Which shard contains this key?" When a client wants to read or write data:
- Query the directory with the key
- Receive the shard location from the directory
- Contact that specific shard to perform the operation
This indirection gives you enormous flexibility. You can:
- Move data between shards without changing partition logic
- Rebalance load by updating directory entries
- Handle hotspots by splitting frequently-accessed keys to different shards
- Support custom partition strategies per key range
Tradeoffs
The directory itself becomes a critical component—if it fails or becomes a bottleneck, your entire system stalls. You'll typically replicate the directory and cache its mappings at clients to mitigate this. The extra lookup adds slight latency compared to direct calculation methods like hash-based partitioning.
Key Takeaway: Directory-based partitioning decouples partition assignment from the partitioning algorithm by maintaining an explicit lookup service, providing maximum flexibility at the cost of added complexity and potential bottlenecks.