Network Partitions and Split-Brain
What You'll Learn
You'll understand how network failures can divide a distributed system into isolated groups, causing nodes to make conflicting decisions about system state.
What is a Network Partition?
A network partition occurs when nodes in a distributed system lose the ability to communicate with each other, typically due to network failures, router crashes, or firewall misconfigurations. The system effectively splits into separate "islands" where nodes within each island can talk to each other, but cannot reach nodes in other islands.
Think of it like a company with offices connected by phone lines. If the lines go down, each office continues working independently—but now they might make contradictory decisions because they can't coordinate.
The Split-Brain Problem
When a partition occurs, each isolated group may believe it's still the authoritative part of the system. This creates split-brain: multiple groups simultaneously acting as "leaders" or making independent decisions about shared data.
Example scenario: A database cluster with three nodes (A, B, C) splits into two groups: {A, B} and {C}. If both groups accept writes:
- Group {A, B} updates a user's email to "new@example.com"
- Group {C} updates the same user's email to "old@example.com"
When the network heals, whose update wins? The system now has inconsistent state.
Why This Matters
Network partitions are inevitable in real distributed systems—networks will fail. Systems must either:
- Choose consistency: Refuse operations during partitions to prevent conflicts
- Choose availability: Accept operations and resolve conflicts later
- Use consensus protocols: Require majority agreement before acting (like requiring 2 of 3 nodes to agree)
Key Takeaway: Network partitions force distributed systems to choose between consistency and availability, making split-brain scenarios a fundamental challenge requiring careful design decisions about conflict handling.