Stateful Async Iterators

Stateful Async Iterators

In this lesson, you will learn how to maintain persistent memory within an asynchronous generator to track progress across sequential data chunks.

The "Memory" Problem

When we create an async function* (async generator), each time we call .next(), the function pauses and resumes. Normally, this is stateless—the generator doesn't "know" what happened in the previous loop unless you tell it. However, in real-world data processing (like reading a large file or streaming paginated API results), you often need to remember how much data you’ve processed so far, or keep a running total.

By defining variables outside the yield loop but inside the generator function, you create a "stateful" iterator. This state persists as long as the iterator is alive.

A Concrete Example: Tracking Progress

Imagine we are processing a stream of data packets, and we want to keep track of a "sequence ID" for every chunk we yield.

async function* sequenceGenerator(dataChunks) {
  let count = 0; // Our internal state

  for (const chunk of dataChunks) {
    count++; // State updates every time we iterate
    
    // Simulate an async operation, like writing to a DB
    await new Promise(resolve => setTimeout(resolve, 100));
    
    yield {
      id: count,
      data: chunk
    };
  }
}

// Usage
const stream = sequenceGenerator(['A', 'B', 'C']);
for await (const item of stream) {
  console.log(`Processed chunk ${item.id}: ${item.data}`);
}

Why This Matters

In this example, the variable count acts as our memory. Even though the generator pauses at yield and waits for the next for await iteration, count remains in memory.

This pattern is incredibly powerful for:

1. Pagination: Tracking the nextPageToken from an API.
2. Buffering: Accumulating small chunks until they reach a certain size before yielding.
3. Progress reporting: Knowing exactly which percentage of a stream has been processed.

By combining closure-based state with async/await, you turn a simple iterator into a smart, context-aware machine that remembers its history as it traverses asynchronous data.

Key Takeaway: By defining variables within the scope of an async generator function, you create persistent state that tracks your progress through an asynchronous data stream across multiple yields.