In this lesson, you will learn how to configure Kafka's retention and compaction policies to balance disk storage constraints with the requirement to maintain historical or stateful data.
When you stream data into a Kafka topic, that data occupies physical disk space on your brokers. If left unchecked, your storage will eventually fill up, leading to broker failures. To manage this, Kafka provides two primary strategies for cleaning up logs: Retention Policies and Log Compaction.
Time and Size-Based Retention
Retention policies are used when you want to treat a Kafka topic as a sliding window of historical events. You can configure these at the topic level using:
retention.ms: The maximum time to keep data (e.g., 7 days).retention.bytes: The maximum size the partition can reach before deleting the oldest segments.
This is ideal for transient telemetry or log data where the "newest" information is the most relevant.
Log Compaction: Keeping the Latest State
Log compaction is a different beast. Instead of deleting data based on age, Kafka ensures that for every key in your topic, the last known value is always retained.
Think of this as a database snapshot of your current state. If you have a stream of user_id updates, compaction removes older entries for the same user_id, keeping only the most recent one. This is crucial for event-sourcing or maintaining a "lookup table" within a Kafka topic. You enable this by setting:
# Configure these properties on your topic
cleanup.policy=compact
delete.retention.ms=86400000
Choosing the Right Strategy
When choosing your policy, consider your downstream consumer requirements:
- Use Retention Policies when your consumers process a continuous stream of events and don't care about state (e.g., streaming logs to an S3 data lake).
- Use Log Compaction when your consumers need to reconstruct the "current state" of an object without replaying the entire history of the topic (e.g., a service starting up and needing to know the latest status of all active orders).
By applying these policies, you move from an "unlimited growth" model to a predictable, self-maintaining storage architecture.
Key Takeaway: Use retention policies to enforce a sliding window for time-series data, and use log compaction to guarantee that the latest state for every unique key is always available for your consumers.