Spark Streaming and Structured Streaming
What you'll learn: How Spark processes real-time data using two different streaming models: micro-batches and continuous processing.
The Core Idea
Apache Spark offers two approaches to stream processing. Spark Streaming (the older API) uses micro-batch processing—it divides the incoming data stream into tiny batches and processes each batch using Spark's existing batch processing engine. Structured Streaming (the newer API) can use either micro-batching or continuous processing for true low-latency streaming.
Micro-Batch Processing
Think of micro-batching like a conveyor belt that stops briefly every few seconds. Spark Streaming collects all events that arrived during a short interval (say, 1-5 seconds), bundles them into a mini-batch, and processes that batch as if it were a static dataset. Once done, it moves to the next batch.
Advantages: Leverages Spark's proven batch processing capabilities, provides exactly-once semantics, and offers good throughput.
Tradeoff: Latency is bounded by the batch interval—you can't get sub-second latencies easily because you're always waiting for the next batch window to close.
Continuous Processing
Structured Streaming's continuous mode processes each record as soon as it arrives, without waiting for batch boundaries. It's like a true assembly line that never stops.
Advantages: Achieves millisecond-level end-to-end latencies, suitable for applications requiring immediate responses.
Tradeoff: More complex to manage, fewer fault-tolerance guarantees (currently at-least-once instead of exactly-once), and less mature than micro-batching.
When to Use Which
Use micro-batch (Structured Streaming default) when you need strong consistency guarantees and can tolerate 1-10 second latencies. Use continuous processing when you need ultra-low latency and can handle at-least-once semantics.
Key Takeaway: Spark Streaming uses micro-batch processing to divide streams into small batches, while Structured Streaming supports both micro-batching and continuous processing for different latency requirements.