Kafka Retention and Storage
What you'll learn: How Kafka stores messages on disk, manages data lifecycle through retention policies, and uses compaction to keep history manageable.
Log-Based Storage Model
Kafka stores messages in immutable, append-only logs on disk. When a producer sends a message to a topic, Kafka appends it to the end of a log segment file. Think of it like a never-ending journal where you only write at the end—never editing previous entries.
Each partition is stored as a sequence of log files. Messages within a partition maintain strict ordering and receive sequential offsets. This design makes Kafka incredibly fast for both writes (always appending) and reads (sequential I/O).
Retention Policies
Kafka doesn't keep messages forever—you control how long data persists using retention policies:
- Time-based retention: Keep messages for a specific duration (e.g., 7 days). After that window, Kafka deletes old log segments regardless of whether consumers read them.
- Size-based retention: Cap total storage per partition (e.g., 1 GB). Once exceeded, the oldest segments are removed.
These policies operate at the topic level, letting different topics have different lifespans. Critical audit logs might retain for 90 days, while high-volume metrics keep only 24 hours.
Log Compaction
Compaction is a special retention mode that keeps only the latest value for each message key. Instead of time-based deletion, Kafka periodically scans partitions and removes older messages with duplicate keys, retaining the most recent update.
This is perfect for maintaining current state—like a user profile database. If user ID 42's email changes three times, compaction ensures you keep only the final version, not the entire history.
Key Takeaway: Kafka's log-based storage combines append-only performance with flexible retention policies (time, size, or compaction) to balance durability, cost, and use-case requirements.