Shard Failure and Partial Outages
You'll learn how a single shard failure limits damage to a subset of users, and strategies to keep systems resilient.
The Blast Radius Problem
When your database is sharded, each shard holds a portion of your total data. If one shard goes down, only the users whose data lives on that specific shard are affected—not your entire user base.
Analogy: Imagine a library system with 10 branches across a city. If one branch closes for repairs, only patrons who belong to that branch lose access to their books. The other 9 branches keep serving their communities normally. This is a partial outage, not a total system failure.
Why This Matters
In a non-sharded (monolithic) database, a single failure takes down everyone. With sharding:
- Limited impact: If you have 8 shards and one fails, roughly 1/8 of users are affected
- Isolated failures: Problems don't cascade to healthy shards
- Continued revenue: Your system still processes 7/8 of transactions
The downside? You now have multiple points of failure instead of one. More shards mean more things that can go wrong.
Resilience Strategies
- Replication per shard: Each shard has its own replicas (covered earlier), so a single server failure doesn't kill the shard
- Monitoring and alerting: Detect shard failures quickly to minimize user impact
- Graceful degradation: Show affected users a "temporarily unavailable" message instead of crashing
- Faster failover: Promote a replica to primary quickly to restore service for that shard's users
Key Takeaway: Sharding converts catastrophic total outages into manageable partial outages affecting only users on the failed shard, making replication per shard essential for true resilience.