In this lesson, you will learn how to choose between batch processing and real-time streaming when moving data from a source system into your data pipeline.
When you connect to a source system (like a customer database or a web server), you must decide how to grab that data. This decision rests on two main patterns: Batch Processing and Real-time Streaming.
1. Batch Processing
Batch processing is like waiting for the mail to arrive once a day. You collect a large "chunk" of data over a specific time interval (e.g., every midnight) and move it all at once.
- Best for: Systems where data doesn't change instantly, or when you have high volumes of data that aren't time-sensitive.
- Example: A nightly job that exports all sales from the previous 24 hours into a data warehouse.
-- Conceptual Batch: Move everything created since the last run
SELECT * FROM orders
WHERE created_at >= '2023-10-26 00:00:00'
AND created_at < '2023-10-27 00:00:00';
2. Real-Time Streaming
Streaming is like a live conversation. Instead of waiting for a batch, data is ingested the moment it is generated. As soon as an event occurs, it is pushed into your pipeline.
- Best for: Situations where split-second decisions matter, such as fraud detection, live user tracking, or monitoring server health.
- Example: Capturing every single "click" a user makes on a website to provide instant product recommendations.
How to choose?
To decide which pattern fits your source system, ask yourself one question: "How quickly does the business need this information to be useful?"
If your business can wait until the morning to see yesterday's report, use Batch. It is simpler, cheaper, and easier to manage. If the business loses money or misses an opportunity because the data is even five minutes old, you need Streaming.
Think of batch as "periodic snapshots" and streaming as a "continuous video feed." By evaluating the source system's constraints and the business's urgency, you can select the pattern that balances performance with cost.
Key Takeaway: Choose batch processing for reliability and simplicity when data latency is acceptable, and choose real-time streaming when immediate visibility into events is mission-critical.