Disaster Recovery Testing

In this lesson, you will learn how to transition from simply having backups to ensuring your business survives a catastrophic outage by simulating failure scenarios.

The Theory: Verify, Don't Just Hope

Having a backup strategy—whether physical (using pg_basebackup) or logical (using pg_dump)—is only half the battle. Disaster Recovery (DR) testing is the practice of proving that your recovery procedures work under pressure. If you haven’t successfully restored your data to a clean environment, you don’t have a backup; you have a leap of faith.

Simulating a Failure

To validate your procedures, you should treat a DR test like a fire drill. The goal is to reach a "Recovery Point Objective" (RPO) and "Recovery Time Objective" (RTO).

1. Isolation: Never run a restore test on your production instance. Provision a temporary, isolated staging server that mimics production hardware configurations.
2. The "Clean Slate" Procedure: If you rely on physical backups for a Postgres cluster, your test should look like this:

   # 1. Stop the standby/test instance
   systemctl stop postgresql
   # 2. Wipe the data directory
   rm -rf /var/lib/postgresql/data/*
   # 3. Restore from your latest physical backup
   pg_basebackup -h backup_server -D /var/lib/postgresql/data -P
   # 4. Configure recovery.conf (or use signal files) to point to WAL archives
   # 5. Start the instance and verify data integrity
   systemctl start postgresql
   

3. Validation: Once the instance is up, run integrity checks. Query key tables to ensure rows exist and, more importantly, check the latest timestamps to see if your point-in-time recovery (PITR) successfully captured the most recent transactions from your WAL archives.

Analyzing the Gaps

During this simulation, document the "time to recovery." If it takes six hours to restore but your business requirement is two, you know your current strategy (e.g., pulling backups over a slow network) is insufficient. You might need to move to local high-speed storage for backups or implement faster streaming replication failovers using Patroni.

Key Takeaway: A disaster recovery plan is only as reliable as your last successful, documented, and timed restoration drill.