Cluster Membership Changes in Raft
What you'll learn: How Raft safely reconfigures a cluster by adding or removing servers without risking split-brain scenarios.
The Configuration Change Problem
In a real-world distributed system, you'll need to add servers (to handle more load) or remove them (for maintenance or downsizing). The naive approach—just switching from the old configuration to the new one—is dangerous. Different servers might switch at different times, creating a brief window where two separate majorities could form, potentially electing two leaders simultaneously. This violates Raft's safety guarantees.
Joint Consensus: The Two-Phase Approach
Raft solves this using joint consensus. Instead of switching directly from old config (C-old) to new config (C-new), it introduces an intermediate transitional configuration (C-old,new) that includes servers from both configurations.
Here's the process:
- Leader receives configuration change request: The current leader appends a special
C-old,newlog entry - Joint consensus phase: While in C-old,new, decisions (elections, commits) require majorities from both C-old AND C-new. A leader must get votes from both groups, and entries commit only when replicated to majorities of both
- Transition to C-new: Once C-old,new commits, the leader appends a C-new entry
- Full transition: After C-new commits, servers in C-old but not C-new can safely shut down
Why This Works
The beauty is that there's never a moment when C-old and C-new can make independent decisions. During the joint phase, every decision needs approval from both groups, preventing split-brain. It's like requiring two different committees to both approve a motion before it passes—neither can act unilaterally.
Key Takeaway: Raft handles cluster membership changes through joint consensus, ensuring safety by requiring agreement from both old and new configurations during the transition period, preventing any possibility of divergent majorities.