Course contentsShow
← System Design
Lesson 745 of 1,91920. Data Pipelines and ETLFree lesson

Exactly-Once Semantics

Achieving exactly-once processing guarantees in streaming requires idempotent operations and transactional coordination.

Exactly-Once Semantics

What you'll learn: How streaming systems achieve exactly-once processing guarantees through idempotent operations and transactional coordination.

The Challenge

In stream processing, delivering a message is one thing—processing it exactly once is much harder. Network failures, crashes, or retries can cause duplicate processing. If you're updating account balances or inventory counts, processing the same event twice creates serious problems.

The Core Solution

Exactly-once semantics (EOS) in streaming requires two coordinated mechanisms:

1. Idempotent Operations
An idempotent operation produces the same result whether executed once or multiple times. Instead of "add 10tobalance,"youwrite"setbalanceto10 to balance," you write "set balance to 110 for transaction ID xyz." If the system retries, the outcome stays correct.

Think of a light switch: flipping it to "on" multiple times doesn't make the room brighter—idempotency ensures repeated actions are safe.

2. Transactional Coordination
The system must atomically commit both the processing result and the consumer offset (the "I've processed this" marker). If either fails, both roll back. This prevents scenarios where you process a message but crash before marking it done, or vice versa.

How It Works Together

When processing an event:

  • Read from the stream
  • Apply idempotent transformations
  • Write results and offset in a single transaction
  • Commit atomically

If anything fails mid-transaction, the system retries from the last committed offset. Because operations are idempotent and commits are atomic, reprocessing produces the correct result without duplicates.

The Tradeoff

EOS adds complexity and latency overhead. You need transactional storage, careful state management, and coordination protocols. Many systems settle for at-least-once delivery with idempotent consumers as a simpler alternative.

Key Takeaway: Exactly-once semantics combines idempotent operations (safe to retry) with transactional coordination (atomic commits) to guarantee each event affects your system precisely once, even during failures.