Log Compaction: Snapshotting
What you'll learn: How Raft prevents logs from growing forever by taking snapshots of committed state.
The Problem: Unbounded Log Growth
In Raft, every state change is recorded as an entry in the replicated log. Over time—days, weeks, months—this log grows without limit. Storing and replaying billions of old entries becomes impractical: disk fills up, recovery takes hours, and new servers take forever to catch up.
The Solution: Snapshots
Snapshotting is Raft's log compaction mechanism. Once the state machine has applied many committed entries, the server takes a snapshot—a compact representation of the current application state at a specific point.
Think of it like checkpointing in a video game: instead of replaying every action from the beginning, you save your progress and start fresh from that save point.
How It Works
- Take a snapshot: Periodically, a server serializes its current state machine state (e.g., a key-value store's entire dataset) to disk
- Record metadata: The snapshot includes the last included index and term—the log position it represents
- Discard old log entries: All entries up to and including that index can now be safely deleted
- Install on lagging followers: If a follower is too far behind, the leader sends the snapshot via
InstallSnapshot RPCinstead of individual log entries
Why It's Safe
The snapshot only includes committed state—entries that have been safely replicated to a majority. Uncommitted entries remain in the log. The snapshot essentially acts as a starting point; new entries continue to build on top of it.
Key Takeaway: Snapshotting allows Raft to discard old log entries by periodically capturing committed state, preventing unbounded log growth while preserving correctness and enabling efficient catch-up for lagging servers.