Schema Evolution in Replicated Systems

In this lesson, you will learn how to safely evolve your database schema across a replicated fleet without triggering downtime or breaking your replication stream.

The Challenge of Distributed Schemas

When you have a primary-replica architecture, your schema changes must be applied in a way that respects the replication protocol. If you run a destructive ALTER TABLE statement on the primary, that command is serialized into the replication stream (e.g., binlog or WAL) and sent to all followers. If the change takes too long or creates a lock, your replicas will lag, potentially leading to service degradation or "replication drift," where the schema states across your nodes diverge.

The "Expand and Contract" Pattern

To maintain high availability, we avoid "all-at-once" changes. Instead, we use a multi-step workflow known as Expand and Contract.

1. Expand (Additive Changes): First, apply schema changes that are backward-compatible. Adding a nullable column or a new table is "safe" because it doesn't break existing application code or the binary structure of data already being replicated.
2. Migrate (Data Synchronization): Update your application logic to dual-write to the old and new structures if you are renaming or splitting columns.
3. Contract (Cleanup): Only once all replicas have successfully processed the schema update and the application is no longer reading from the deprecated fields, do you run the DROP or cleanup commands.

Example: Safely Renaming a Column

If you try to rename a column with ALTER TABLE users RENAME COLUMN old_name TO new_name, your replication stream will break if your application code still references old_name. Instead, do this:

-- 1. Add new column (Add only)
ALTER TABLE users ADD COLUMN new_name VARCHAR(255);

-- 2. Update code to write to both columns and read from 'old_name'
-- 3. Backfill data from 'old_name' to 'new_name' in the background.

-- 4. Finally, update code to read from 'new_name'
-- 5. Drop the old column
ALTER TABLE users DROP COLUMN old_name;

By decoupling the database metadata change from the application logic, you ensure that replicas never encounter a state where the data they receive violates the expectations of the code currently running on those nodes.

Key Takeaway: Always favor additive schema changes that remain backward-compatible, ensuring that every replication step is a non-breaking transition for your entire distributed system.