In this lesson, you will learn how to implement fencing mechanisms to neutralize the risk of data divergence when network partitions cause a cluster to split.
The Split-Brain Problem
A "split-brain" occurs when a network partition isolates nodes in a distributed system, causing two or more partitions to believe they are the legitimate leader. Without proper safeguards, both sides might accept conflicting writes, leading to permanent data divergence that is often impossible to reconcile automatically. Since we’ve already explored the limitations of the CAP theorem and distributed consensus, you know that safety is the primary concern during these partitions.
Fencing Mechanisms
To prevent this, we implement fencing. Fencing acts as a "kill switch" to ensure that an old, isolated leader cannot commit writes to the storage layer even if it still believes it is the leader.
- STONITH (Shoot The Other Node In The Head): This is a hardware-level fencing mechanism. If the cluster management layer detects a node is unresponsive, it triggers a power-off command via a Power Distribution Unit (PDU) or management controller (like IPMI). By physically cutting power to the suspected node, we guarantee it cannot perform conflicting writes.
- Resource Fencing (Lock-based): In systems where we cannot power-cycle nodes, we use locks. When a new leader takes over, it must acquire a "lease" or a lock on a shared, highly available resource (like a Zookeeper node or a shared disk). The lock includes an epoch number (or generation clock).
-- Conceptual implementation of a fenced write
-- The storage engine rejects writes from older epochs
UPDATE data_table
SET value = 'new_val'
WHERE epoch_id >= CURRENT_LEADER_EPOCH;
When a write reaches the storage layer, the storage engine checks the epoch_id. If a node attempts to write with an epoch_id lower than the current established leader’s, the write is rejected. This ensures that even if an old leader survives the partition, it is effectively "fenced out" of the state machine.
Fencing forces a system to prioritize safety over availability, ensuring that only the recognized authority can commit state changes.
Key Takeaway: Fencing prevents split-brain data corruption by either physically disabling isolated nodes or enforcing epoch-based write rejection to ensure only one partition holds the authoritative lock.