Streaming Large Data Exports

Streaming Large Data Exports

In this lesson, you will learn how to serve massive datasets to users by streaming them directly from a data source to the HTTP response, keeping your server's memory usage near zero regardless of the file size.

The Memory Bottleneck

When you need to export a large dataset (like a million-row CSV), a common mistake is to fetch all the data from your database, store it in an array in memory, and then send it. If you have 50 concurrent users requesting a 500MB report, your server will quickly crash with an OutOfMemory error. Even if one request succeeds, the overhead of allocating that massive buffer puts unnecessary pressure on the Garbage Collector.

The Streaming Pattern

Instead of "loading" data, you "pipe" it. By treating the HTTP response as a Writable stream, you can push data chunks as soon as they are retrieved from your source. This ensures that only a tiny fraction of the data exists in your application's RAM at any given moment.

Here is how you would bridge a database cursor (like from MongoDB or PostgreSQL) directly to an HTTP response:

const { pipeline } = require('stream');

// Suppose 'dbCursor' is a readable stream of data rows
// 'res' is the Express or Node.js HTTP response object
pipeline(
  dbCursor,
  jsonTransformer, // Custom transform stream to format data as CSV/JSON
  res,
  (err) => {
    if (err) console.error('Streaming failed:', err);
  }
);

Why This Matters

By using pipeline, you gain two critical benefits:

1. Backpressure management: If the client's network is slow, the stream will automatically pause reading from the database so your server buffer doesn't overflow.
2. Fixed Memory Footprint: Whether you are exporting 10MB or 10GB, your server's memory consumption remains flat because it never holds more than a few kilobytes of the file in memory at once.

Key Takeaway: Never load large exports into memory; always pipe data from your source directly to the response object to keep your server lightweight and responsive.