Manual Failover Procedures
What you'll learn: How to orchestrate and validate manual failovers when automatic mechanisms aren't suitable or have failed.
When Manual Failover Makes Sense
While automatic failover (lesson 2483) handles most scenarios, manual failover is essential when:
- You need to perform planned maintenance on the primary database
- Automatic failover has malfunctioned or is disabled
- You want controlled testing without production risk
- The situation requires human judgment (e.g., ambiguous failure signals)
Think of it like a pilot switching to manual controls when autopilot isn't appropriate—you need clear procedures and verification steps.
Core Manual Failover Steps
1. Pre-Failover Verification
- Confirm replica health and replication lag (from lesson 2477)
- Check that the target replica has up-to-date data
- Notify stakeholders of planned downtime window
- Document current primary identifier
2. Initiate Failover
# Example promotion command (cloud-agnostic concept)
promote-replica --instance=replica-01 --force
3. Post-Failover Validation
- Verify new primary is accepting write traffic
- Confirm DNS or endpoint updates have propagated
- Test application connectivity
- Monitor replication from old primary (if still running)
4. Rollback Considerations If issues arise, you need:
- Quick rollback path: Can you demote and re-promote?
- Data divergence check: Have writes occurred on both nodes?
- Split-brain prevention: Ensure old primary is truly offline
The Human Factor
Manual failover demands clear runbooks (lesson 2441), rehearsed procedures, and calm decision-making under pressure.
Key Takeaway: Manual failover gives you control and predictability for planned events, but requires rigorous verification steps and well-documented rollback procedures to prevent data loss or split-brain scenarios.