Designing for High Availability (HA)

In this lesson, you will learn how to define the operational guardrails of High Availability (HA) by quantifying your system’s tolerance for data loss and downtime.

High Availability is not a binary state; it is a measurable service level. To design for it effectively, we define two critical success criteria: Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

Defining RTO and RPO

• RTO (Recovery Time Objective): This is the duration of time within which a business process must be restored after a disaster or failure. If your shard goes offline, RTO is how long you can afford to wait before the partition-aware client library successfully redirects traffic to a healthy replica.
• RPO (Recovery Point Objective): This measures the maximum tolerable period in which data might be lost from an IT service due to a major incident. It is essentially the "age" of the data you are willing to lose, governed by your transaction log replay lag and the asynchronous nature of your cross-cluster replication.

Practical Application

When configuring your distributed architecture, these objectives dictate your replication strategy. For instance, if your RPO is near zero, you cannot rely solely on asynchronous write-ahead logging (WAL) replicas; you must implement synchronous replication, which impacts latency but ensures durability.

Consider this configuration snapshot for a partition:

-- Conceptual HA Policy for a Distributed Partition
ALTER TABLE orders_partition_1 SET (
    ha_rto_seconds = 30,    -- Max downtime for automatic failover
    ha_rpo_bytes = 0,       -- Synchronous commitment required
    replication_mode = 'sync' 
);

If your distributed query planner detects a network partition, the system must trigger a failover protocol. If the time taken to elect a new leader exceeds your RTO, your design has failed to meet its availability promise. If the failover results in the loss of un-replicated WAL entries, you have exceeded your RPO.

When you design for HA, you are essentially balancing the trade-off between strict data consistency (lower RPO) and continuous system availability (lower RTO). Your architectural choices—from consistency tuning to replication factor optimization—must be strictly aligned with these two KPIs.

Key Takeaway: High Availability success is defined by RTO (how fast you recover) and RPO (how much data you can afford to lose), which together dictate your replication and failover architecture.