Course contentsShow
← Computer Science
Lesson 2689 of 2,87260. Distributed Data and CommunicationFree lesson

Directory-Based Partitioning

Maintaining a lookup service that maps keys to shards for flexible partition assignment.

Directory-Based Partitioning

You'll learn: How to use a lookup service to map keys to shards, enabling flexible partition assignment that can be changed without reorganizing all your data.

The Core Idea

In previous partitioning strategies like range-based or hash-based partitioning, the algorithm itself determines which shard holds a given key. Directory-based partitioning takes a different approach: it maintains a separate lookup table (the "directory") that explicitly stores which partition holds each key or key range.

Think of it like a phone book for your data. Instead of calculating "Person A lives at address hash(A) % num_servers," you simply look up "Where does Person A live?" in your directory and get back "Server 3."

How It Works

The directory service acts as a central registry that answers the question: "Which shard contains this key?" When a client wants to read or write data:

  1. Query the directory with the key
  2. Receive the shard location from the directory
  3. Contact that specific shard to perform the operation

This indirection gives you enormous flexibility. You can:

  • Move data between shards without changing partition logic
  • Rebalance load by updating directory entries
  • Handle hotspots by splitting frequently-accessed keys to different shards
  • Support custom partition strategies per key range

Tradeoffs

The directory itself becomes a critical component—if it fails or becomes a bottleneck, your entire system stalls. You'll typically replicate the directory and cache its mappings at clients to mitigate this. The extra lookup adds slight latency compared to direct calculation methods like hash-based partitioning.

Key Takeaway: Directory-based partitioning decouples partition assignment from the partitioning algorithm by maintaining an explicit lookup service, providing maximum flexibility at the cost of added complexity and potential bottlenecks.