In this lesson, you will learn how to configure fence mechanisms to safely isolate non-responsive nodes in a Galera cluster to prevent data inconsistency during a split-brain event.
The Split-Brain Problem
A "split-brain" occurs when a network partition causes a cluster to divide into two or more sub-groups that cannot communicate. If both sides believe they are the legitimate cluster, they might both accept writes, leading to irreversible data divergence. In Galera, we mitigate this via quorum-based logic, but "fencing" takes it a step further by physically or logically disconnecting a rogue node from the network or storage before it can cause harm.
Implementing Fencing Mechanisms
To safely isolate a non-responsive node, we implement fencing mechanisms, often referred to as STONITH ("Shoot The Other Node In The Head"). In a high-availability database ecosystem, this is typically handled by an external resource manager (like Pacemaker) or by configuring the cluster to enter a non-operational state when it detects it has lost its majority.
If a node identifies that it is no longer part of the primary component, we configure the wsrep_provider_options to force the node to stop accepting transactions immediately.
-- Example: Configuring Galera to reject traffic when disconnected
SET GLOBAL wsrep_provider_options = 'pc.ignore_sb=no;pc.bootstrap=no;';
When a node loses its heartbeat (as discussed in your previous lesson on Replication Heartbeat Monitoring), the cluster quorum mechanism should trigger a fence. If you are using a proxy layer like ProxySQL, you can integrate fencing by instructing the proxy to immediately shunt all traffic away from the isolated node as soon as it enters a non-primary state:
- Detection: The monitoring tool detects the lack of heartbeats.
- Isolation: The node is removed from the load balancer rotation via
mysql_serverstable updates. - Fencing: If the node continues to hold resources, an automation script (via SSH or an IPMI command) initiates a
service mysql stopto ensure the node cannot process stale write requests.
By automating the isolation of nodes that lose quorum, you ensure that even if the network fluctuates, only the healthy, synchronized majority of the cluster remains active.
Key Takeaway: Fencing prevents split-brain disasters by automatically isolating non-responsive nodes from both client traffic and the underlying storage layer, ensuring that only the authorized quorum can commit transactions.