In this lesson, you will learn how to determine when to stick with a continuous stream and when to pivot to batch processing for data transformations.
While streams are excellent for memory efficiency and low-latency processing, they aren't always the perfect tool for every job. As a rule of thumb, streaming is best when the data is flowing indefinitely or when you need to start outputting results before the entire input is received. Batching becomes the superior choice when your transformation requires a "global" view of the data that a single chunk cannot provide.
When to Switch to Batching
Consider a transformation where you need to calculate the median or standard deviation of a dataset. Since these operations require knowledge of the full distribution, streaming individual items won't work—you would have to buffer the entire stream in memory to perform the calculation, which negates the benefits of streaming.
If you find yourself using a Transform stream that buffers every incoming chunk until the _flush method is called, you are essentially "faking" a stream. At that point, it is cleaner and more performant to use a batching approach.
Illustrative Example
If you are calculating a simple rolling average, a stream is perfect:
// Streaming: Efficient, low memory
stream.pipe(new MovingAverageTransform());
However, if you need to calculate a complex metric like a sort or a global median:
// Batching: Necessary for operations requiring full context
const data = await readAllRecords(source);
const result = calculateComplexMetric(data);
await writeToDestination(result);
If the dataset is small enough to fit in memory, the overhead of managing a complex stream pipeline—with all its associated backpressure and state-tracking requirements—can be replaced by a simple, synchronous batch process. If the dataset is too large for memory, you should use a disk-based batching strategy or a database-level aggregation instead of forcing the data through a Node.js stream pipeline that is secretly waiting for the final piece of the puzzle.
Key Takeaway: Use streams for continuous, chunk-based processing; switch to batching when your transformation logic requires a complete view of the dataset or access to global state that cannot be calculated incrementally.