Failover Orchestration Strategies

In this lesson, you will learn how to design custom orchestration logic to safely promote a standby node to primary status while ensuring data integrity.

The Problem: Split-Brain Scenarios

Failover orchestration isn't just about starting a service; it's about preventing a "split-brain" scenario where two nodes believe they are the primary, leading to data divergence. When your primary node fails, you need a deterministic script that ensures the old primary is fenced off before the new one starts accepting writes.

Anatomy of a Controlled Failover Script

Effective orchestration requires a "Check-Promote-Update" flow. Since you are already familiar with Patroni (Lesson 13) and Replication Lag (Lesson 17), your custom orchestration should leverage those metrics as gates.

Here is a conceptual bash-based orchestration pattern:

#!/bin/bash
# 1. Verify Primary health
if ! psql -h primary_node -c "SELECT 1" > /dev/null; then
    # 2. Check if standby is ready (minimal lag)
    LAG=$(psql -h standby_node -t -c "SELECT pg_last_wal_receive_lsn() - pg_last_wal_replay_lsn()")
    if [ "$LAG" -lt 1048576 ]; then # 1MB threshold
        # 3. Trigger promotion
        ssh standby_node "pg_ctl promote -D /var/lib/postgresql/data"
        # 4. Update routing (e.g., PgBouncer config or VIP)
        update_routing_to_new_primary.sh
    else
        echo "Lag too high, manual intervention required."
    fi
fi

Critical Orchestration Components

1. Fencing (STONITH): Before promoting a standby, the orchestrator must confirm the old primary is unreachable or stopped (Shoot The Other Node In The Head). If you promote a node while the old primary still accepts writes, you will have conflicting timelines.
2. Read-Only Verification: Once the standby is promoted, it must change its internal state. You can enforce this by ensuring your pg_hba.conf and connection pool settings (via PgBouncer) are updated immediately to point writes toward the new node.
3. Observation vs. Action: Your orchestration logic should rely on heartbeat signals rather than simple connection timeouts to avoid "flapping" during transient network spikes.

By building these logic gates into your failover scripts, you transition from reactive recovery to a predictable, automated HA lifecycle.

Key Takeaway: A robust failover strategy must prioritize consistency by verifying replica lag and enforcing fencing before promoting a standby to primary.