Course contentsShow
← System Design
Lesson 1477 of 1,91936. Data Partitioning Strategies in DepthFree lesson

Directory Service Architecture

Building and scaling the metadata service that maintains partition mappings.

Directory Service Architecture

What you'll learn: How to design and scale the metadata service that tracks which partition lives where—the "phone book" of your distributed data system.

The Core Problem

When you partition data across many servers, something must answer the question: "Where does partition X live right now?" This is the directory service—a centralized or semi-centralized registry mapping partition IDs to physical nodes.

Think of it like a library card catalog. The books (data) live on shelves (nodes), but you need the catalog (directory) to find them. Without it, clients would have to ask every server, "Do you have partition 42?"

Architecture Components

The directory service typically has three layers:

1. Storage Layer

The actual mapping data: { partition_id → [node_addresses] }. Often stored in a strongly-consistent system like etcd, ZooKeeper, or Consul to avoid stale routing.

2. Query Layer

API endpoints clients use: getPartitionLocation(key) or listPartitionsOnNode(nodeId). Must be fast—this sits in the hot path of every read/write.

3. Update Layer

Handles partition moves, splits, merges. When a range split occurs (from lesson 1475), the directory must atomically update mappings so clients never see inconsistency.

Scaling Challenges

Reads dominate writes (mappings change rarely), so cache aggressively at clients. Use versioned maps so clients know when their cache is stale. For massive scale, shard the directory itself by partition-ID ranges—your directory becomes a mini-partitioned database.

Key Takeaway:
The directory service is the authoritative source-of-truth for partition placement; design it for high availability and aggressive caching, since every data operation depends on it.