Handling Replica Failures in Routing
You'll learn how to detect when read replicas go down and automatically route traffic away from them to keep your application running smoothly.
The Challenge
Even with well-configured read replicas, hardware fails, networks partition, and databases crash. When a replica becomes unavailable, your routing layer must detect the failure and stop sending queries there—otherwise users experience timeouts and errors.
How It Works
Think of replicas like checkout lanes at a grocery store. If a cashier suddenly leaves their station, the manager needs to notice quickly and redirect customers to open lanes. Your routing system does the same:
1. Health Checks
The load balancer or connection pool regularly pings each replica with simple queries (like SELECT 1). If a replica doesn't respond within a timeout window, it's marked unhealthy.
2. Removal from Rotation
Once detected as failed, the replica is immediately removed from the pool of available read targets. New queries bypass it entirely.
3. Fallback Strategy
Traffic redistributes across remaining healthy replicas. If all replicas fail, you can fall back to reading from the primary database (accepting the increased load) or return degraded service errors, depending on your requirements.
4. Retry and Recovery
The system continues health-checking the failed replica. Once it responds successfully again, it's gradually reintroduced to the rotation—sometimes with reduced traffic initially to verify stability.
Example Flow
Request → Router checks replica pool
→ Replica A: healthy ✓
→ Replica B: timeout ✗ (removed)
→ Route to Replica A or Primary
Key Takeaway: Detecting replica failures through health checks and automatically removing them from routing prevents user-facing errors and maintains system availability even when individual replicas fail.