Course contentsShow
← System Design
Lesson 718 of 1,91919. Event Streaming PlatformsFree lesson

Exactly-Once Semantics (EOS)

How Kafka achieves exactly-once delivery using idempotent producers, transactions, and consumer read isolation levels.

Exactly-Once Semantics (EOS)

What you'll learn: How Kafka guarantees that messages are processed exactly once, even during failures.

The Challenge

By default, Kafka offers at-least-once delivery—messages might be duplicated during retries or consumer crashes. At-most-once risks data loss. Exactly-once means each message affects the system precisely once: no duplicates, no losses, even when producers retry or consumers crash.

How Kafka Achieves EOS

Kafka combines three mechanisms to deliver exactly-once semantics:

1. Idempotent Producers
Enable enable.idempotence=true. Kafka assigns each producer a unique ID and sequences every message. If a producer retries sending a message (thinking it failed), the broker detects the duplicate sequence number and discards it silently. Think of it like writing a check with a check number—the bank won't cash #1234 twice.

2. Transactions
Producers can wrap multiple messages (even to different topics/partitions) in a transaction. Either all messages commit or none do. Kafka tracks transaction state and coordinates with consumers to ensure atomicity. This is crucial when one input message generates multiple output messages.

3. Consumer Read Isolation Levels
Consumers set isolation.level to:

  • read_uncommitted (default): sees all messages immediately
  • read_committed: only sees messages from committed transactions

Using read_committed prevents consumers from processing data that might get rolled back, ensuring they only act on finalized writes.

The Full Picture

EOS typically means: idempotent producer + transactional writes + read_committed consumers. This guarantees end-to-end exactly-once processing from input topic → processing logic → output topic, even across failures.

Key Takeaway:
Kafka's exactly-once semantics combine idempotent producers (prevent duplicate sends), transactions (atomic multi-message writes), and read isolation levels (consumers only see committed data) to guarantee each message is processed exactly once.