Streaming Large File Uploads

In this lesson, you will learn how to architect a system that consumes multi-gigabyte files via HTTP requests without exhausting your server’s RAM.

The Memory Bottleneck

When a user uploads a 5GB file, a naive implementation might attempt to buffer that file into memory or store it as a temporary object. In Node.js, this leads to an ERR_STRING_TOO_LONG or a process crash due to the V8 heap limit. Even if you don't buffer the whole file, loading chunks too quickly can overwhelm the event loop if the consumer (like a database or disk writer) is slower than the uploader.

The Pipeline Architecture

To handle large uploads safely, you must treat the incoming HTTP request as a Readable stream and pipe it directly to a destination. By using the pipeline helper, you ensure that backpressure is automatically managed between the incoming stream and the storage layer.

Here is how you architect this flow:

const { pipeline } = require('stream');
const fs = require('fs');

function handleUpload(req, res) {
  // 'req' is our readable stream of the incoming file
  const writeStream = fs.createWriteStream('./uploads/large-file.bin');

  // pipeline connects the streams and handles cleanup/backpressure
  pipeline(req, writeStream, (err) => {
    if (err) {
      console.error('Upload failed:', err);
      res.statusCode = 500;
      return res.end('Upload failed');
    }
    res.end('Upload successful');
  });
}

Why this works

1. Constant Memory Footprint: The req stream reads only small chunks (governed by the high water mark) into memory at a time.
2. Backpressure Sensitivity: If writeStream cannot keep up (e.g., the disk is slow), the pipeline helper signals the req stream to pause reading from the socket. This forces the client to slow down their upload speed rather than crashing your server.
3. Event-Driven Cleanup: If the client disconnects halfway through, pipeline automatically destroys the writeStream to prevent dangling file handles, ensuring your system remains stable.

Key Takeaway: To process massive files safely, always pipe your readable request stream directly to a writable destination, allowing the underlying stream machinery to throttle the data flow based on your system's current capacity.