Directory-Based Sharding

In this lesson, you will learn how to implement a lookup service to manage dynamic partition mappings, allowing your database to scale beyond the rigid constraints of mathematical hash functions.

The Limitation of Fixed Sharding

In previous lessons, we covered Range-Based and Hash-Based Sharding. These strategies rely on a deterministic algorithm (e.g., hash(user_id) % N) to find data. While efficient, they suffer from a "rebalancing nightmare": if you add a new node, you must move a massive percentage of your data to maintain the mapping. Directory-Based Sharding solves this by decoupling the data location from the key itself.

The Lookup Service Architecture

Instead of calculating the destination, you maintain a central "Directory" (or Shard Map). When the application needs to read or write, it first queries this metadata store to see which shard currently owns the requested key.

Think of it as a routing table in a network. The directory keeps an entry for every partition:

-- Conceptual Shard Directory Table
CREATE TABLE shard_map (
    partition_id INT PRIMARY KEY,
    node_address VARCHAR(255),
    status VARCHAR(20) -- e.g., 'active', 'migrating'
);

-- Mapping for a specific range or key group
CREATE TABLE key_lookup (
    key_range_start INT,
    partition_id INT
);

Why Use a Directory?

1. Dynamic Rebalancing: If a node becomes a "hotspot," you can move a specific range to a new node, update the shard_map, and the application automatically routes traffic to the new location without needing to change your hashing logic.
2. Heterogeneous Hardware: You can assign larger, high-performance shards to nodes with more RAM or faster NVMe drives by simply updating the mapping, rather than being forced to treat all nodes as identical slices.
3. Complex Migration: During a data move, the Directory can mark a shard as migrating, allowing the system to proxy reads to the source while writes are directed to the destination during synchronization.

The overhead is an extra network hop to the directory, which is usually mitigated by caching the map at the application layer and using a Gossip Protocol (Lesson 8) to invalidate caches when mappings change.

Key Takeaway: Directory-Based Sharding introduces a level of indirection that replaces rigid, computed mappings with a flexible lookup table, enabling granular, zero-downtime data rebalancing.