Course contentsShow
← Cloud and Infrastructure
Lesson 2610 of 2,98956. Data Pipelines and StreamingFree lesson

Kafka Retention and Storage

Understand Kafka's log-based storage, retention policies, and compaction.

Kafka Retention and Storage

What you'll learn: How Kafka stores messages on disk, manages data lifecycle through retention policies, and uses compaction to keep history manageable.


Log-Based Storage Model

Kafka stores messages in immutable, append-only logs on disk. When a producer sends a message to a topic, Kafka appends it to the end of a log segment file. Think of it like a never-ending journal where you only write at the end—never editing previous entries.

Each partition is stored as a sequence of log files. Messages within a partition maintain strict ordering and receive sequential offsets. This design makes Kafka incredibly fast for both writes (always appending) and reads (sequential I/O).

Retention Policies

Kafka doesn't keep messages forever—you control how long data persists using retention policies:

  • Time-based retention: Keep messages for a specific duration (e.g., 7 days). After that window, Kafka deletes old log segments regardless of whether consumers read them.
  • Size-based retention: Cap total storage per partition (e.g., 1 GB). Once exceeded, the oldest segments are removed.

These policies operate at the topic level, letting different topics have different lifespans. Critical audit logs might retain for 90 days, while high-volume metrics keep only 24 hours.

Log Compaction

Compaction is a special retention mode that keeps only the latest value for each message key. Instead of time-based deletion, Kafka periodically scans partitions and removes older messages with duplicate keys, retaining the most recent update.

This is perfect for maintaining current state—like a user profile database. If user ID 42's email changes three times, compaction ensures you keep only the final version, not the entire history.

Key Takeaway: Kafka's log-based storage combines append-only performance with flexible retention policies (time, size, or compaction) to balance durability, cost, and use-case requirements.