In this lesson, you will learn how to prevent memory overflows by respecting stream backpressure when piping data between asynchronous sources and destinations.
The Problem: Memory Bloat
In Node.js, streams process data piece-by-piece. However, readable.pipe(writable) is only efficient if the destination can keep up with the source. If you have a high-speed file reader (source) piping into a slow network socket (destination), the source will keep reading data into your RAM. If the destination can't drain that data fast enough, your application’s memory usage will balloon until the process crashes with an Out of Memory error.
Managing the Flow
Backpressure is the mechanism that signals the source to "slow down." When the writable stream's buffer exceeds its highWaterMark, it returns false to the .write() call. This tells the upstream source to pause until the destination is ready to accept more data.
The modern, recommended way to handle this—and ensure robust error propagation—is using stream.pipeline. It automatically manages the piping process and cleans up resources if one stream fails.
const { pipeline } = require('stream');
const fs = require('fs');
const zlib = require('zlib');
// pipeline manages backpressure automatically.
// If the destination slows down, the source is automatically paused.
pipeline(
fs.createReadStream('large-data.json'),
zlib.createGzip(),
fs.createWriteStream('data.json.gz'),
(err) => {
if (err) {
console.error('Pipeline failed:', err);
} else {
console.log('Pipeline finished successfully.');
}
}
);
Why Manual Piping is Risky
If you manually listen to data events and call dest.write(), you must check the return value of write(). If it returns false, you must call source.pause() and wait for the drain event on the destination before calling source.resume(). This manual logic is error-prone and often leads to subtle bugs. By using stream.pipeline (or the newer stream/promises API), you offload this complexity to Node.js’s internals, ensuring your system remains responsive even under heavy load.
Key Takeaway: Always use stream.pipeline instead of manual .pipe() to ensure backpressure is correctly handled and resources are cleaned up during failures.