Leader Election and Failover
You'll learn how systems automatically promote a backup replica to leader when the primary database fails.
The Core Challenge
In single-leader replication, all writes flow through one primary node. But what happens when that leader crashes? Without intervention, your system can't accept writes—or worse, multiple nodes might think they're the leader (split-brain). Leader election is the process of choosing a new leader from the surviving replicas, and failover is the orchestrated transition to that new leader.
How Leader Election Works
Think of it like a presidential succession plan. When the leader becomes unresponsive (detected via health checks and timeouts), the remaining replicas must agree on who takes over. This requires consensus—a majority vote among nodes to prevent two leaders emerging simultaneously.
Common approaches:
- Manual failover: Operators promote a replica by hand (slow, error-prone)
- Automatic failover: The system uses algorithms like Raft or Paxos to elect a new leader based on criteria like "most up-to-date replica" or "lowest node ID"
The Split-Brain Problem
Without consensus, two replicas might both declare themselves leader during a network partition. Now you have conflicting writes that are nearly impossible to reconcile. Quorum-based voting prevents this: a node can only become leader if it receives votes from a majority of nodes.
Typical Failover Steps
- Detect leader failure (missed heartbeats)
- Trigger election among replicas
- Promote winner to leader
- Redirect client writes to new leader
- Let old leader rejoin as replica (if it recovers)
Key Takeaway: Leader election uses consensus protocols to automatically promote a replica to leader during failures, with quorum requirements preventing dangerous split-brain scenarios.