Distributed Backup and Recovery

In this lesson, you will learn how to orchestrate snapshot strategies that maintain global state consistency across multiple distributed partitions.

The Challenge of Distributed Snapshots

When your data is spread across multiple shards, taking a backup isn't as simple as running a pg_dump or mysqldump on a single node. Because shards operate independently, a standard backup would result in a "skewed" view: Shard A might capture data at time $T_1$, while Shard B captures it at $T_2$. If your application relies on referential integrity or cross-shard transactions, this temporal drift renders your entire backup useless for point-in-time recovery.

The Chandy-Lamport Algorithm Concept

To achieve a consistent global state, we use the principle of a Distributed Snapshot. The goal is to record a "global state" where every shard agrees on a consistent boundary, usually defined by a transaction ID or a Global Log Sequence Number (LSN).

Instead of stopping the entire cluster, we use a Marker Message approach:

1. A coordinator sends a "snapshot marker" to all shards.
2. Upon receiving the marker, a shard flushes its current memory buffer to stable storage and records its current LSN.
3. If any cross-shard communication (like a distributed transaction) occurs during this window, the shard records the incoming/outgoing messages to ensure dependencies are preserved.

Practical Implementation

In a sharded environment, you don't necessarily need to implement the full algorithm from scratch; you can leverage the Global Transaction Coordinator.

-- Conceptual: Triggering a globally consistent snapshot
-- 1. Coordinator initiates a global barrier at LSN 9999
SELECT pg_create_global_snapshot('backup_2023_10_27', 9999);

-- 2. Each shard executes locally at its own pace:
-- Shard A (Range 1-1000):
COPY (SELECT * FROM orders WHERE lsn <= 9999) TO '/backups/shard_a_9999.csv';

-- Shard B (Range 1001-2000):
COPY (SELECT * FROM orders WHERE lsn <= 9999) TO '/backups/shard_b_9999.csv';

By ensuring every shard halts or logically partitions its writes at the exact same global marker, you guarantee that no "orphan" records exist in Shard B that refer to a state in Shard A that hasn't been captured yet.

Key Takeaway: Consistent distributed backups require a shared logical clock or global transaction ID to synchronize the state across partitions, ensuring that the recovered dataset represents a single, valid point in time.