In this lesson, you will learn how to manage the unbounded growth of distributed consensus logs to ensure your cluster remains performant and recoverable.
The Challenge of Unbounded Logs
In consensus protocols like Raft or Paxos, nodes maintain an append-only log of every state-changing operation. As the system processes millions of requests, this log grows indefinitely. This creates two critical problems:
- Disk Exhaustion: Storing the entire history consumes excessive storage.
- Recovery Latency: If a node crashes, it must replay the entire log from the beginning of time to reconstruct its state, which could take hours.
Snapshots: Freezing the State
To solve this, we implement snapshots. Instead of keeping every log entry, we periodically capture the current state of the database (the result of applying all logs up to index N) and save it as a serialized image on disk.
Once a snapshot is successfully persisted, all log entries prior to index N are considered redundant. We can safely truncate the log, keeping only the entries that occurred after the snapshot.
How it looks in practice:
Imagine a key-value store where the log contains:
SET A=1SET B=2SET A=3
If we snapshot after operation 3, our state is {"A": 3, "B": 2}. We discard logs 1, 2, and 3. The new log starts at index 4.
Log Compaction
While snapshots replace the entire state, Log Compaction (specifically used in systems like Kafka or some distributed KV stores) focuses on redundant key-value updates. If a log contains multiple updates for the same key, compaction runs a background process to retain only the latest value for each key, effectively "collapsing" the log while preserving the final outcome.
By combining these two, we keep the log size manageable and ensure that when a node restarts, it only needs to load the most recent snapshot and replay a tiny fraction of the recent log to reach a consistent state.
Key Takeaway: Snapshots and log compaction transform an infinitely growing sequence of operations into a compact, fixed-size state, drastically reducing recovery time and storage overhead.