Course contentsShow
← System Design
Lesson 711 of 1,91919. Event Streaming PlatformsFree lesson

Message Retention and Log Segments

Understand Kafka's append-only log structure, retention policies (time or size-based), and how log segments enable compaction.

Message Retention and Log Segments

What you'll learn: How Kafka manages message storage through an append-only log structure, retention policies, and log segments.

The Append-Only Log Structure

Kafka stores messages in an append-only log for each partition. Think of it like a diary that only lets you write new entries at the end—you never erase or modify what you've already written. This design makes writes incredibly fast (just append to the file) and allows consumers to read from any point in time by specifying an offset.

Retention Policies

Kafka doesn't keep messages forever—that would fill up your disk quickly! Instead, you configure retention policies that automatically clean up old data:

  • Time-based retention: Keep messages for a specified duration (e.g., 7 days). After that, they're eligible for deletion regardless of whether anyone has read them.
  • Size-based retention: Keep the most recent messages up to a certain total size (e.g., 1 GB per partition). When the limit is exceeded, the oldest messages are removed.

This is different from traditional message queues where messages disappear after consumption. In Kafka, retention is independent of whether consumers have read the messages.

Log Segments

Physically, each partition's log is divided into segments—smaller, manageable files on disk. Think of segments like chapters in that diary. Kafka writes to the active segment and periodically rotates to a new one.

Segments enable efficient operations:

  • Deletion: When messages expire per retention policy, Kafka simply deletes entire old segment files (fast!)
  • Compaction: Kafka can compact segments by keeping only the latest value for each key, useful for maintaining state snapshots

Key Takeaway: Kafka's append-only log with configurable retention (time or size) and segment-based storage enables efficient, durable message streaming with the ability to replay historical data while managing disk usage.