Directory-Based Sharding
What you'll learn: How to use a centralized lookup service to route data requests to the correct shard, offering flexibility at the cost of an extra dependency.
The Core Idea
In directory-based sharding, you maintain a separate lookup service (the "directory") that maps each piece of data (or its key) to a specific shard. Instead of using a mathematical formula like hash-based or range-based sharding, your application queries this directory to find out where the data lives, then fetches it from that shard.
Think of it like a library card catalog: instead of guessing which shelf a book is on, you check the catalog first. The catalog tells you exactly where to go.
How It Works
- Application receives a request with a key (e.g.,
user_id=12345) - Query the directory service: "Where is
user_id=12345stored?" - Directory responds: "Shard C"
- Application connects to Shard C and retrieves the data
The directory is essentially a mapping table or configuration store (could be a database, Redis, ZooKeeper, etcd, etc.) that stores:
Key → Shard
user_123 → shard_A
user_456 → shard_B
order_789 → shard_C
Trade-offs
Advantages:
- Flexibility: You can move data between shards by updating the directory—no rehashing needed
- Custom logic: You can use business rules, not just hash functions
Disadvantages:
- Extra lookup overhead: Every request requires two hops (directory, then shard)
- Single point of failure: The directory itself must be highly available
- Scalability bottleneck: The directory can become a performance choke point
Key Takeaway: Directory-based sharding offers flexible, centralized routing by maintaining a lookup service, but introduces latency and a critical dependency that must be carefully managed.