Course contentsShow
← System Design
Lesson 719 of 1,91919. Event Streaming PlatformsFree lesson

Kafka Transactions

Atomic writes across multiple partitions and topics using transactional producers and isolation for read-committed consumers.

Kafka Transactions

What you'll learn: How Kafka enables atomic, all-or-nothing writes across multiple partitions and topics using transactional producers.

The Problem Without Transactions

Imagine you're building an order system where one event should update inventory (Topic A), create a billing record (Topic B), and log the order (Partition 3 of Topic C). If your producer crashes after writing only the first two messages, you're left with inconsistent data—inventory updated but no order logged. This breaks data integrity.

How Kafka Transactions Work

Kafka transactions allow a producer to write multiple messages to different topics/partitions as an atomic unit—either all messages commit successfully, or none do. The producer marks the start of a transaction, publishes messages, and then commits or aborts the entire batch.

Under the hood, Kafka coordinates this using:

  • A transactional ID assigned to the producer
  • A transaction coordinator (a broker managing transaction state)
  • Markers written to the log indicating transaction boundaries (commit/abort)

Read-Committed Isolation

On the consumer side, you can configure isolation.level=read_committed. These consumers only see messages from committed transactions. Aborted messages remain invisible, even though they physically exist in the log. This prevents consumers from processing partial, incomplete transactions.

Think of it like a bank transfer: debit one account, credit another—both must happen or neither should. Kafka transactions guarantee this atomicity across distributed topics.

Tradeoffs

Transactions add latency and complexity. You must handle producer restarts carefully (reusing the same transactional ID for exactly-once guarantees). But when correctness matters more than raw throughput, transactions are essential.

Key Takeaway: Kafka transactions enable atomic, all-or-nothing writes across multiple partitions and topics, with read-committed consumers seeing only fully committed message sets—essential for maintaining data integrity in complex event flows.