Course contentsShow
← System Design
Lesson 713 of 1,91919. Event Streaming PlatformsFree lesson

Kafka's Write Path and Durability

Explore how Kafka persists messages to disk, uses page cache for performance, and configures acks for durability guarantees.

Kafka's Write Path and Durability

What you'll learn: How Kafka efficiently persists messages to disk while balancing performance and durability guarantees.

When a producer sends a message to Kafka, understanding the write path helps you configure the right balance between speed and safety. Kafka's approach is surprisingly clever: it writes sequentially to disk (which is fast) and leverages the operating system's page cache as a buffer.

The Write Path

When a message arrives at the leader broker for a partition:

  1. Write to page cache: Kafka appends the message to an in-memory buffer (the OS page cache) immediately
  2. Async flush to disk: The OS eventually flushes data from page cache to physical disk in the background
  3. Replication: The leader forwards the message to follower replicas based on your configuration

Think of the page cache like a notebook you jot ideas into quickly, knowing someone will later transcribe them carefully into a permanent ledger. This two-tier approach gives you both speed and durability.

Durability Configuration: The acks Setting

Producers control durability with the acks parameter:

  • acks=0: Fire-and-forget. Don't wait for any acknowledgment (fastest, least durable)
  • acks=1: Wait for leader to write to its page cache (balanced)
  • acks=all (or -1): Wait for leader AND all in-sync replicas to acknowledge (slowest, most durable)

With acks=all, even if the leader crashes before flushing to disk, at least one follower has the message in its page cache, preventing data loss.

Performance Insight

Sequential disk writes combined with page cache make Kafka exceptionally fast—often faster than random-access databases. The OS optimizes page cache flushes, batching writes efficiently.

Key Takeaway: Kafka achieves high throughput by writing sequentially to page cache first, then asynchronously flushing to disk, while the acks setting lets you trade latency for durability guarantees.