Sharding and Database Scaling
You'll learn: How to distribute billions of URL mappings across multiple database nodes using hash-based sharding to handle massive scale.
The Scale Challenge
A URL shortener serving billions of shortened links cannot rely on a single database node—it will hit CPU, memory, and I/O limits. Even with read replicas handling queries, the master database becomes a write bottleneck. The solution? Sharding: horizontally partitioning your data across multiple independent database nodes.
Hash-Based Sharding for URLs
Use the short code itself as the sharding key. Apply a hash function (like MD5 or MurmurHash) to the short code, then use modulo arithmetic to determine which shard owns that URL:
shard_id = hash(short_code) % num_shards
Why this works:
- Even distribution: Hash functions scatter keys uniformly across shards
- Predictable lookups: Given a short code, you always know which shard to query
- No hotspots: Unlike range-based partitioning, celebrity URLs don't concentrate on one shard
Implementation Details
Each shard is a fully independent database storing a subset of URLs. Your application layer maintains a shard map (which shard IDs exist and their connection strings). When creating a URL with short code aB3xK, hash it to determine the target shard, then write there. On redirect, repeat the hash to find the correct shard for lookup.
Trade-off: Adding shards requires rebalancing data (though consistent hashing from earlier lessons can minimize this).
Key Takeaway: Sharding URL mappings by hashing the short code distributes billions of records evenly across multiple databases, eliminating single-node bottlenecks and enabling horizontal scaling for both reads and writes.