Directory Service Architecture
What you'll learn: How to design and scale the metadata service that tracks which partition lives where—the "phone book" of your distributed data system.
The Core Problem
When you partition data across many servers, something must answer the question: "Where does partition X live right now?" This is the directory service—a centralized or semi-centralized registry mapping partition IDs to physical nodes.
Think of it like a library card catalog. The books (data) live on shelves (nodes), but you need the catalog (directory) to find them. Without it, clients would have to ask every server, "Do you have partition 42?"
Architecture Components
The directory service typically has three layers:
1. Storage Layer
The actual mapping data: { partition_id → [node_addresses] }. Often stored in a strongly-consistent system like etcd, ZooKeeper, or Consul to avoid stale routing.
2. Query Layer
API endpoints clients use: getPartitionLocation(key) or listPartitionsOnNode(nodeId). Must be fast—this sits in the hot path of every read/write.
3. Update Layer
Handles partition moves, splits, merges. When a range split occurs (from lesson 1475), the directory must atomically update mappings so clients never see inconsistency.
Scaling Challenges
Reads dominate writes (mappings change rarely), so cache aggressively at clients. Use versioned maps so clients know when their cache is stale. For massive scale, shard the directory itself by partition-ID ranges—your directory becomes a mini-partitioned database.
Key Takeaway:
The directory service is the authoritative source-of-truth for partition placement; design it for high availability and aggressive caching, since every data operation depends on it.