Understanding the Micro-Batch Execution Model

In this lesson, you will learn how Spark Structured Streaming bridges the gap between high-throughput batch processing and low-latency streaming by utilizing the micro-batch execution model.

Bridging Batch and Streaming

When we talk about stream processing, the most common challenge is reconciling the speed of incoming events with the efficiency of distributed computing. Spark Structured Streaming addresses this by treating a continuous stream as a series of small, incremental batch jobs.

Instead of processing a single row at a time—which can be resource-intensive due to the overhead of task scheduling—the engine collects incoming data over a short, configurable interval (e.g., 500 milliseconds). These collected events are grouped into a "micro-batch."

The Execution Loop

The execution model follows a consistent, cyclical pattern:

1. Source Discovery: The engine queries the streaming source (like Kafka) to identify the new range of offsets since the last micro-batch.
2. Plan Execution: Spark constructs a query plan for this specific batch. Because it is essentially a standard batch job, it leverages the same optimized catalyst optimizer and physical execution engine used in traditional Spark SQL.
3. Commit: Once the transformation is applied and the data is written to the sink, the engine commits the offsets, ensuring that the system knows exactly where to resume for the next cycle.

This approach provides a "best of both worlds" scenario. It allows you to use familiar DataFrame/Dataset APIs while maintaining the high throughput and fault tolerance characteristic of Spark’s distributed architecture.

// The stream is processed in short, discrete intervals
val query = streamingDataFrame
  .writeStream
  .format("parquet")
  .option("checkpointLocation", "/path/to/checkpoints")
  .trigger(Trigger.ProcessingTime("1 second")) // Defines the micro-batch cadence
  .start()

By batching data, the system reduces the pressure on the scheduler and allows for more effective resource utilization across the cluster. While it introduces a slight latency equal to the batch duration, it ensures that your stream processing remains stable, scalable, and easy to monitor using standard batch processing tools.

Key Takeaway: The micro-batch execution model converts a continuous flow of data into discrete, manageable batches, allowing the engine to leverage highly optimized batch processing techniques while providing the low-latency guarantees required for streaming.