Course contentsShow
← System Design
Lesson 1541 of 1,91937. Real-World System: URL ShortenerFree lesson

Sharding and Database Scaling

Shard URL mappings by hash of short code across multiple database nodes to handle billions of URLs.

Sharding and Database Scaling

You'll learn: How to distribute billions of URL mappings across multiple database nodes using hash-based sharding to handle massive scale.

The Scale Challenge

A URL shortener serving billions of shortened links cannot rely on a single database node—it will hit CPU, memory, and I/O limits. Even with read replicas handling queries, the master database becomes a write bottleneck. The solution? Sharding: horizontally partitioning your data across multiple independent database nodes.

Hash-Based Sharding for URLs

Use the short code itself as the sharding key. Apply a hash function (like MD5 or MurmurHash) to the short code, then use modulo arithmetic to determine which shard owns that URL:

shard_id = hash(short_code) % num_shards

Why this works:

  • Even distribution: Hash functions scatter keys uniformly across shards
  • Predictable lookups: Given a short code, you always know which shard to query
  • No hotspots: Unlike range-based partitioning, celebrity URLs don't concentrate on one shard

Implementation Details

Each shard is a fully independent database storing a subset of URLs. Your application layer maintains a shard map (which shard IDs exist and their connection strings). When creating a URL with short code aB3xK, hash it to determine the target shard, then write there. On redirect, repeat the hash to find the correct shard for lookup.

Trade-off: Adding shards requires rebalancing data (though consistent hashing from earlier lessons can minimize this).

Key Takeaway: Sharding URL mappings by hashing the short code distributes billions of records evenly across multiple databases, eliminating single-node bottlenecks and enabling horizontal scaling for both reads and writes.