In this lesson, you will learn how to design streaming interfaces to process massive datasets without overwhelming your application's memory.
When you query a database for thousands or millions of records, attempting to load them all into an array at once will cause your Node.js process to crash due to "heap out of memory" errors. While basic pagination (using LIMIT and OFFSET) works for user interfaces, it is inefficient for data-intensive retrieval tasks because the database must scan and skip rows repeatedly, causing high disk I/O costs.
Instead, we use streaming interfaces. By leveraging Node.js Readable streams, we can process records one by one as they arrive from the database connection. This keeps your memory usage constant, regardless of whether you are processing 100 rows or 10 million.
Example: Streaming Records
Instead of db.query('SELECT * FROM users') (which returns a giant array), we request a stream:
const query = db.query('SELECT * FROM users');
const stream = query.stream();
stream.on('data', (row) => {
// Process each row individually
console.log(`Processing user: ${row.email}`);
});
stream.on('end', () => {
console.log('Finished processing all records.');
});
By processing each row as it arrives in the stream's data event, your application only ever keeps a tiny fraction of the result set in RAM at any given moment. This approach is essential for background tasks like generating exports, running data migrations, or performing bulk analytics where you don't need the entire set in memory simultaneously.
When designing these interfaces, ensure your query is ordered (e.g., ORDER BY id) to maintain a deterministic flow of data. If you are dealing with truly massive datasets, consider combining this with Keyset Pagination—instead of skipping rows with OFFSET, you fetch the next batch of records based on the last seen id. This prevents the performance degradation that occurs as your database search progresses deeper into the table.
Key Takeaway: Never load large result sets into memory; use Node.js streams to process data record-by-record to keep your application stable and efficient.