In this lesson, you will learn how to use watermarks to balance the trade-off between waiting for late-arriving data and keeping your stream processing pipelines responsive.
The Challenge of Time
In real-world streaming, data rarely arrives in the exact order it was generated. Networks lag, mobile devices go offline, and distributed systems introduce jitter. When we perform windowed operations—like calculating the average number of user clicks in 5-minute intervals—we face a dilemma: if we stop processing a window as soon as the clock hits the end of that interval, we might miss data that was delayed in transit.
What is a Watermark?
A watermark is a threshold that tells the streaming engine: "I am willing to wait X amount of time for late data before I consider this window closed."
Think of it as a grace period. If you define a watermark of 10 minutes, the engine tracks the maximum event time it has observed so far. It then expects that any data arriving with a timestamp older than (Max Observed Time - 10 minutes) is too late to be included in our windowed results. Once the watermark passes the end of a window, the engine clears the state for that window, effectively "closing the books" and releasing memory.
Configuring in Spark
In Spark Structured Streaming, you apply this via the withWatermark method. This informs the engine how to track lateness based on a timestamp column in your data.
# Assuming 'event_time' is the timestamp of the actual event
streaming_df = spark.readStream.table("events") \
.withWatermark("event_time", "10 minutes") \
.groupBy(window("event_time", "5 minutes")) \
.count()
By setting the threshold to "10 minutes," you are telling Spark that it can safely drop or ignore any events that arrive more than 10 minutes after the maximum observed event time. This prevents the state store from growing infinitely and ensures that your streaming job doesn't hold onto memory for windows that will never receive updates again.
Key Takeaway: A watermark creates a sliding boundary that manages state memory by defining exactly how long the system should wait for late-arriving events before finalizing a window.