Course contentsShow
← System Design
Lesson 242 of 1,9197. Database Scaling: ShardingFree lesson

Directory-Based Sharding

Maintain a lookup service that maps keys to shards, enabling flexible but centralized routing.

Directory-Based Sharding

What you'll learn: How to use a centralized lookup service to route data requests to the correct shard, offering flexibility at the cost of an extra dependency.


The Core Idea

In directory-based sharding, you maintain a separate lookup service (the "directory") that maps each piece of data (or its key) to a specific shard. Instead of using a mathematical formula like hash-based or range-based sharding, your application queries this directory to find out where the data lives, then fetches it from that shard.

Think of it like a library card catalog: instead of guessing which shelf a book is on, you check the catalog first. The catalog tells you exactly where to go.


How It Works

  1. Application receives a request with a key (e.g., user_id=12345)
  2. Query the directory service: "Where is user_id=12345 stored?"
  3. Directory responds: "Shard C"
  4. Application connects to Shard C and retrieves the data

The directory is essentially a mapping table or configuration store (could be a database, Redis, ZooKeeper, etcd, etc.) that stores:

Key       → Shard
user_123  → shard_A
user_456  → shard_B
order_789 → shard_C

Trade-offs

Advantages:

  • Flexibility: You can move data between shards by updating the directory—no rehashing needed
  • Custom logic: You can use business rules, not just hash functions

Disadvantages:

  • Extra lookup overhead: Every request requires two hops (directory, then shard)
  • Single point of failure: The directory itself must be highly available
  • Scalability bottleneck: The directory can become a performance choke point

Key Takeaway: Directory-based sharding offers flexible, centralized routing by maintaining a lookup service, but introduces latency and a critical dependency that must be carefully managed.