Configuring Output Sinks and Modes

Select appropriate output modes like append, update, or complete to control how processed results are written to external storage systems.

In this lesson, you will learn how to choose the correct output mode to control how your processed streaming data is persisted into an external sink.

When working with Structured Streaming, simply defining a transformation isn't enough; you must explicitly tell the system how to handle new results. The "Output Mode" acts as a bridge between your continuous processing logic and your storage system (like a Data Lake or a SQL database), defining exactly what data should be written each time a trigger fires.

There are three primary modes you will use:

1. Append Mode

This is the default and most common mode. Only newly appended rows since the last trigger are written to the sink. This is ideal for scenarios where you are writing logs or events to a table that only grows, such as an audit trail or an append-only transaction log. Because previous results are never modified, this mode is highly efficient.

2. Update Mode

Use this when you have performed a stateful aggregation (like a running count or sum) and you only want to output the rows that have changed. If a record has been updated, only the updated row is sent to the sink. This is perfect for syncing a database table where you want to maintain a "current state" view without re-writing the entire history every time.

3. Complete Mode

In this mode, the entire updated result table is written to the sink every time there is a trigger. This is useful for summary dashboards where you want the external system to always hold the absolute latest total of your calculations. Note that this mode is generally only supported with aggregations.

# Example: Writing an aggregated stream to a sink
streaming_df.writeStream \
    .outputMode("update") \
    .format("delta") \
    .option("checkpointLocation", "/path/to/checkpoints") \
    .start("/path/to/output")

Choosing the right mode is a balance between performance and the requirements of your downstream consumers. If you select a mode that doesn't align with your transformation (like attempting Append on an aggregation without watermarking), the system will alert you to avoid data corruption or logical errors.

Key Takeaway: Output modes control whether you write only new data (Append), only changed rows (Update), or the full result set (Complete), allowing you to tailor your stream's final delivery to your specific storage requirements.