Achieving Exactly-Once Processing Semantics

In this lesson, you will learn how to unify checkpointed state and transactional sinks to ensure each event affects your final output exactly once, even during failures.

The Challenge of Duplication

In a distributed streaming system, "Exactly-Once" doesn't mean the system never fails; it means that when a failure occurs, the state of the system is restored as if the failure never happened. If a pipeline processes a batch of data, crashes, and restarts, it must ensure that records already written to the external database aren't written a second time. Achieving this requires a coordinated hand-off between your internal stream processor and the external storage layer.

The Two-Phase Solution

To achieve this, we combine two pillars: Checkpointed State and Transactional Sinks.

1. Checkpointed State: Your streaming engine (like Flink) periodically saves the "position" of your source and the contents of your internal state operators to durable storage.
2. Transactional Sinks: The output connector must support transactions. When a checkpoint starts, the sink opens a transaction. All data processed between checkpoints is written to the database but kept in a "pending" or uncommitted state.

Only when the entire checkpoint completes successfully does the system "commit" the transaction. If the system crashes mid-processing, the transaction is aborted or rolls back, and upon restart, the system resumes from the last successful checkpoint, effectively "forgetting" the partial, uncommitted data.

Conceptual Workflow

Think of it like writing a final exam in pencil with an eraser. You perform your work (state processing) and tentatively write your answers (transactional sink). You only commit to the final version (transaction commit) once the exam supervisor confirms the checkpoint has been safely recorded:

// Conceptual flow of transactional sinks
val stream = env.addSource(kafkaSource)
  .map(processData)
  .addSink(new TransactionalDatabaseSink()) 

// The Sink logic:
// 1. On Checkpoint Start: Open transaction
// 2. Process events: Write to DB (uncommitted)
// 3. On Checkpoint Success: Commit transaction

By ensuring that the commit to the external world is tied to the completion of the internal state snapshot, you eliminate duplicate side effects.

Key Takeaway: Exactly-once processing is achieved by linking the commitment of output data to the completion of distributed checkpoints, ensuring that data is only persisted to external sinks once the processing state is safely acknowledged.