Configuring Two-Phase Commit Sinks

In this lesson, you will learn how to implement two-phase commit (2PC) sinks to ensure your streaming pipeline maintains exactly-once delivery guarantees when writing to external transactional systems.

The Problem: Partial Writes

When a Flink job writes to an external sink (like a database or message queue), the sink might fail midway through a checkpoint. If the sink has already written some data from that checkpoint but the Flink job crashes, you risk duplicate data or partial commits upon restart. Simply relying on Flink's internal checkpointing isn't enough because the external system isn't "aware" of Flink’s state.

The Solution: Two-Phase Commit

The Two-Phase Commit protocol bridges this gap by decoupling the writing of data from the committing of that data. It functions in two distinct stages:

1. The Pre-commit Phase: When Flink triggers a checkpoint, the sink begins a transaction in the external system and writes all data received during the current interval. Crucially, it does not commit the transaction yet; it simply buffers the work and "pre-commits" the transaction in the external system.
2. The Commit Phase: Once Flink successfully completes the global checkpoint (meaning all operators have finished their snapshot), the JobManager notifies the sink. Only then does the sink finalize and "commit" the transaction in the external system.

Conceptual Implementation

To implement this, your sink must be able to handle state associated with these transactions. You generally manage this by overriding a TwoPhaseCommitSinkFunction. The flow looks like this:

// Logic flow for a 2PC Sink
public void snapshotState(FunctionSnapshotContext context) {
    // 1. Pre-commit: Open transaction and write data
    // 2. Save the transaction handle in Flink's state
}

public void notifyCheckpointComplete(long checkpointId) {
    // 3. Commit: Retrieve the handle and finalize the transaction
}

If the job fails before the notifyCheckpointComplete signal arrives, the external transaction will eventually time out or be aborted, ensuring no partial or corrupted data remains in the sink. This provides the atomicity required for true exactly-once processing.

Key Takeaway: Two-Phase Commit sinks ensure data integrity by delaying the finalization of transactions in external systems until Flink confirms that the entire checkpoint cycle has been successfully persisted.