In this lesson, you will learn how to process massive files without crashing your application by using Node.js streams to handle data in small, manageable chunks.
The Memory Trap of "Read-All"
When you use functions like fs.readFile, Node.js attempts to load the entire contents of a file into your application's heap memory at once. If your file is 2GB and your server only has 1GB of memory allocated, your process will crash with an "out of memory" error. This is a common bottleneck when dealing with high-volume data, logs, or file uploads.
Processing with Streams
Streaming treats a file as a continuous flow of data rather than a single static block. Instead of pulling the whole file into memory, a ReadStream emits small chunks (typically 64KB buffers) one after another. You process the chunk, let it be garbage collected, and move on to the next.
This approach is highly memory-efficient because your memory footprint stays constant, regardless of whether the file size is 1MB or 1TB.
const fs = require('fs');
// Create a readable stream for a large file
const readStream = fs.createReadStream('huge-data.csv');
const writeStream = fs.createWriteStream('processed-data.csv');
// Pipe the read stream into the write stream
// This moves data chunk-by-chunk automatically
readStream.pipe(writeStream);
readStream.on('error', (err) => console.error('Streaming error:', err));
In the example above, pipe() acts as the bridge. It reads a chunk from the file and writes it to the destination, managing "backpressure"—a mechanism that prevents the read stream from overwhelming the write stream if the destination is slower than the source. By using streams, you avoid loading the entire content into the heap, ensuring your application remains responsive and stable even under heavy I/O load.
Key Takeaway: By using streams to process files in small, iterative buffers rather than reading the entire file into memory, you maintain a predictable and minimal heap usage regardless of total file size.