Distinguishing Batch Processing from Stream Processing

Evaluate the architectural differences between processing bounded datasets and handling continuous, unbounded streams of incoming data events.

In this lesson, you will learn to distinguish between processing static, bounded datasets and operating on continuous, unbounded event streams.

The Batch Paradigm: Bounded Data

Batch processing treats data as a finite collection. You wait for a specific volume or a temporal trigger (like a nightly job) to start processing a known, immutable set of records. Because the dataset is bounded, the engine can look at the entire set before computing a result. This is ideal for heavy aggregations or complex historical reports where data completeness is guaranteed before computation begins.

The Stream Paradigm: Unbounded Data

Stream processing, by contrast, treats data as a continuous flow. The dataset is unbounded; there is no definitive "end" to the stream. Instead of waiting for a batch to finish, the engine processes events as they arrive. This architectural shift is essential for low-latency requirements, such as fraud detection or real-time monitoring, where waiting for a batch to complete would render the information obsolete.

Architectural Contrast

Consider a simplified view of how these models handle a user event, such as a purchase:

Batch: The system periodically reads a file of 1 million purchases, sorts them, and updates a dashboard once an hour.
```
-- Batch: Querying a static snapshot
SELECT user_id, SUM(amount) FROM purchases_table GROUP BY user_id;
```
Stream: The system ingests each purchase event individually, updating a state store in memory so the dashboard reflects the change in milliseconds.
```
-- Stream: Processing an incoming event
INSERT INTO user_spend_totals 
SELECT user_id, amount FROM purchase_stream;
```

In batch processing, the data is at rest in storage, and the processing job moves to the data. In stream processing, the data is in motion, and the processing logic stays resident, waiting to apply logic to each incoming record. Architecting for streams requires managing state (the running totals) and time (handling out-of-order events), which are complexities absent in the simpler, periodic batch model.

Key Takeaway: Batch processing handles finite, static data in cycles, while stream processing reacts to infinite, continuous events in real-time.