Log Compaction
What you'll learn: How Kafka's log compaction keeps only the latest value for each key, enabling powerful event sourcing and changelog patterns.
The Core Idea
Log compaction is a retention policy where Kafka retains only the most recent value for each message key within a partition, rather than deleting messages after a time-based or size-based threshold. Think of it like a database changelog: you don't need every update to a record, just the final state.
How It Works
When log compaction runs, Kafka scans through older log segments and removes all but the latest message for each key. Messages with null values (tombstones) eventually cause the key to be removed entirely—this is how you "delete" in a compacted topic.
Without compaction:
key=user123, value={name: "Alice", email: "a@example.com"}
key=user123, value={name: "Alice", email: "alice@example.com"}
key=user123, value={name: "Alice Smith", email: "alice@example.com"}
After compaction:
key=user123, value={name: "Alice Smith", email: "alice@example.com"}
Use Cases
Event Sourcing: Store every state change as an event. Compaction ensures you can always rebuild current state without unbounded storage.
Changelog Semantics: Kafka can mirror database changes. Each row's current state is preserved, making it perfect for replication or cache warming.
Configuration Management: Store service configs keyed by service name. Consumers always get the latest config when they start up.
Why It Matters
Log compaction transforms Kafka from a temporary message queue into a durable state store. New consumers can read the entire compacted log and reconstruct the current state of the world without needing a separate database.
Key Takeaway: Log compaction retains only the latest value per key, making Kafka suitable for event sourcing, changelogs, and stateful streaming applications that need both history and bounded storage.