In this lesson, you will learn how to process massive data streams using for-await-of loops to keep your application’s memory usage low and stable.
The Memory Problem
When dealing with large files or infinite data streams—like processing a log file with millions of entries—trying to load everything into memory at once with Promise.all or a standard array will crash your Node.js process. You need a way to process one "chunk" of data at a time, waiting for the current task to finish before asking for the next.
Processing with for-await-of
Async Iterators allow you to iterate over data that arrives asynchronously. Instead of fetching the entire dataset into an array, you use an iterator to "pull" data one piece at a time. The for-await-of loop is the standard syntax for consuming these asynchronous sequences.
Consider a stream that fetches data from an API one page at a time:
async function processDataStream(iterator) {
// The loop waits for each chunk to resolve before moving to the next
for await (const chunk of iterator) {
console.log(`Processing chunk of size: ${chunk.length}`);
// Memory is freed after each loop iteration
await saveToDatabase(chunk);
}
console.log("Stream processing complete.");
}
Why this is efficient
In the code above, the for-await-of loop pauses execution at the await keyword. Because you are not storing the entire stream in a variable, the Node.js garbage collector can clear each chunk from memory as soon as saveToDatabase finishes.
If you were to try to use a standard .forEach or map with this, the loop would finish instantly, firing off thousands of requests simultaneously and exhausting your system's RAM. By using for-await-of, you create a "backpressure" mechanism: the program only requests the next piece of data when it is actually ready to handle it. This keeps your application footprint predictable, regardless of whether you are processing ten items or ten billion.
Key Takeaway: Use for-await-of to process data streams one chunk at a time, ensuring your application remains memory-efficient and responsive under heavy loads.