Course contentsShow
← System Design
Lesson 227 of 1,9196. Database Scaling: Read ReplicasFree lesson

Handling Replica Failures in Routing

Detecting unavailable replicas, removing them from rotation, and falling back to primary or other replicas.

Handling Replica Failures in Routing

You'll learn how to detect when read replicas go down and automatically route traffic away from them to keep your application running smoothly.

The Challenge

Even with well-configured read replicas, hardware fails, networks partition, and databases crash. When a replica becomes unavailable, your routing layer must detect the failure and stop sending queries there—otherwise users experience timeouts and errors.

How It Works

Think of replicas like checkout lanes at a grocery store. If a cashier suddenly leaves their station, the manager needs to notice quickly and redirect customers to open lanes. Your routing system does the same:

1. Health Checks

The load balancer or connection pool regularly pings each replica with simple queries (like SELECT 1). If a replica doesn't respond within a timeout window, it's marked unhealthy.

2. Removal from Rotation

Once detected as failed, the replica is immediately removed from the pool of available read targets. New queries bypass it entirely.

3. Fallback Strategy

Traffic redistributes across remaining healthy replicas. If all replicas fail, you can fall back to reading from the primary database (accepting the increased load) or return degraded service errors, depending on your requirements.

4. Retry and Recovery

The system continues health-checking the failed replica. Once it responds successfully again, it's gradually reintroduced to the rotation—sometimes with reduced traffic initially to verify stability.

Example Flow

Request → Router checks replica pool
         → Replica A: healthy ✓
         → Replica B: timeout ✗ (removed)
         → Route to Replica A or Primary

Key Takeaway: Detecting replica failures through health checks and automatically removing them from routing prevents user-facing errors and maintains system availability even when individual replicas fail.