Course contentsShow
← System Design
Lesson 720 of 1,91919. Event Streaming PlatformsFree lesson

Log Compaction

How compacted topics retain only the latest value per key, enabling event sourcing and changelog semantics.

Log Compaction

What you'll learn: How Kafka's log compaction keeps only the latest value for each key, enabling powerful event sourcing and changelog patterns.

The Core Idea

Log compaction is a retention policy where Kafka retains only the most recent value for each message key within a partition, rather than deleting messages after a time-based or size-based threshold. Think of it like a database changelog: you don't need every update to a record, just the final state.

How It Works

When log compaction runs, Kafka scans through older log segments and removes all but the latest message for each key. Messages with null values (tombstones) eventually cause the key to be removed entirely—this is how you "delete" in a compacted topic.

Without compaction:

key=user123, value={name: "Alice", email: "a@example.com"}
key=user123, value={name: "Alice", email: "alice@example.com"}
key=user123, value={name: "Alice Smith", email: "alice@example.com"}

After compaction:

key=user123, value={name: "Alice Smith", email: "alice@example.com"}

Use Cases

Event Sourcing: Store every state change as an event. Compaction ensures you can always rebuild current state without unbounded storage.

Changelog Semantics: Kafka can mirror database changes. Each row's current state is preserved, making it perfect for replication or cache warming.

Configuration Management: Store service configs keyed by service name. Consumers always get the latest config when they start up.

Why It Matters

Log compaction transforms Kafka from a temporary message queue into a durable state store. New consumers can read the entire compacted log and reconstruct the current state of the world without needing a separate database.

Key Takeaway: Log compaction retains only the latest value per key, making Kafka suitable for event sourcing, changelogs, and stateful streaming applications that need both history and bounded storage.