Applying Sliding Windows for Continuous Updates

In this lesson, you will learn how to create overlapping time windows to calculate smooth, continuous moving averages on high-velocity data streams.

The Concept of Sliding Windows

While Tumbling Windows divide your data into discrete, non-overlapping buckets (e.g., 10:00–10:05, 10:05–10:10), Sliding Windows allow these buckets to overlap. A sliding window is defined by two parameters: the window duration (how far back the window looks) and the slide interval (how often the window updates).

Imagine a stock price dashboard. If you want to see a 10-minute moving average that updates every 1 minute, a tumbling window would force you to wait for a full 10 minutes for a new data point. A sliding window, however, gives you a new calculation every minute, always considering the most recent 10 minutes of activity. This provides a "smooth" experience for the end-user, as the metrics evolve gradually rather than jumping abruptly at the end of each fixed interval.

Implementing in a Streaming Context

To define this in a streaming dataframe, you specify both the window length and the slide interval within your grouping logic. If you don’t define a slide interval, it defaults to the window duration (becoming a tumbling window).

# Calculating a 10-minute moving average that slides every 1 minute
sliding_avg = df.groupBy(
    window(col("event_time"), "10 minutes", "1 minute"), 
    col("sensor_id")
).agg(avg("temperature").alias("moving_avg"))

Why Use Sliding Windows?

1. Frequent Updates: You gain visibility into data trends more often than a non-overlapping window would allow.
2. Smoothing: By calculating the average over a longer duration that shifts incrementally, you reduce the "noise" created by sudden spikes or dips in high-velocity data.
3. Continuous Perception: It creates a fluid user experience for real-time monitoring, as the analytical output reflects the most current state of the stream with every small progression of time.

Key Takeaway: Sliding windows provide a persistent, "moving" view of your data by using overlapping time intervals to ensure that analytics update frequently and smoothly as new events arrive.