Course contentsShow
← System Design
Lesson 258 of 1,9197. Database Scaling: ShardingFree lesson

Resharding and Data Migration

Learn the operational complexity of moving data when adding or removing shards from the cluster.

Resharding and Data Migration

What you'll learn: How changing your shard count or rebalancing data across shards becomes one of the most operationally complex challenges in distributed databases.

The Challenge of Growth

Once you've sharded your database, your system isn't frozen in time. You might need to add more shards as data grows, remove shards to consolidate, or rebalance data when hotspots emerge (like the Celebrity Problem). This process is called resharding, and it involves migrating massive amounts of live data while your application continues running.

Why It's Operationally Complex

Imagine reorganizing a library's books across multiple buildings while people are still checking books in and out. That's resharding.

Key challenges:

  • Downtime avoidance: Users expect 24/7 availability
  • Data consistency: Writes happening during migration must not be lost
  • Performance impact: Migration consumes I/O, CPU, and network resources
  • Dual-read/write periods: Applications may need to check both old and new locations temporarily
  • Rollback complexity: If migration fails halfway, how do you recover?

Common Migration Approaches

  1. Stop-the-world migration: Shut down, move data, restart (rarely acceptable)
  2. Dual-write phase: Write to both old and new shards while copying historical data
  3. Shadow traffic: Route reads to old shard, writes to both, gradually shift reads
  4. Consistent hashing helps: Minimizes data movement when adding/removing nodes

If your shard key is immutable and you chose a good distribution strategy (like consistent hashing), resharding becomes less painful but never easy.

Key Takeaway: Resharding is the "moving day" for distributed data—plan for it early by choosing immutable shard keys, using consistent hashing, and building migration tooling before you desperately need it.