Course contentsShow
← System Design
Lesson 712 of 1,91919. Event Streaming PlatformsFree lesson

Log Compaction

Learn how log compaction retains only the latest value per key, enabling Kafka to serve as a durable state store.

Log Compaction

What you'll learn: How Kafka uses log compaction to keep only the most recent value for each key, transforming a stream into a durable state store.


The Problem with Infinite Retention

In lesson 711, you learned that Kafka retains messages based on time or size limits. But what if you want to keep the latest state of each entity forever without accumulating all historical updates? This is where log compaction comes in.

What Is Log Compaction?

Log compaction is a retention policy where Kafka keeps only the latest value for each message key within a partition. Instead of deleting old messages based on time, Kafka discards outdated values while preserving the most recent update for every key.

Plain English Analogy

Imagine a filing cabinet where each folder represents a customer ID. Every time a customer updates their address, you add a new paper to their folder. With log compaction, a janitor periodically visits and removes all old address papers, keeping only the newest one per customer. The cabinet stays current without growing infinitely.

How It Works

When compaction runs on a partition:

  1. Kafka scans the log segments
  2. For each message key, it identifies the latest value
  3. Older values with the same key are discarded
  4. The compacted log contains only the most recent state per key

Use Cases

  • Change Data Capture (CDC): Maintain the current state of database rows
  • User profiles: Keep latest user settings without full history
  • Configuration stores: Retain current config values, not all changes

Key Takeaway:
Log compaction transforms Kafka topics into durable, replay-able state stores by retaining only the latest value per key, enabling stream-based applications to rebuild their state from the log.