Implementing a Custom Readable Stream

In this lesson, you will learn how to build a custom Readable stream that generates data on-demand rather than dumping it all into memory at once.

The Pull-Based Philosophy

In Node.js, a Readable stream is essentially a "pull" mechanism. Instead of the stream forcing data onto the consumer, the consumer asks for data when it is ready. To implement this, we extend the Readable class and override the _read() method.

The _read(size) method is our signal from the system. When it is called, the system is saying: "I have room in my internal buffer; please provide me with more chunks." If we have data ready, we call this.push(chunk). If we run out of data, we call this.push(null) to signal the end of the stream.

A Concrete Example: The Infinite Counter

Imagine we want a stream that outputs incrementing numbers. We shouldn't generate all the numbers at once (that would crash our memory). We only generate them when the consumer pulls.

const { Readable } = require('stream');

class CounterStream extends Readable {
  constructor(limit) {
    super();
    this.limit = limit;
    this.count = 1;
  }

  _read() {
    if (this.count > this.limit) {
      return this.push(null); // Signal EOF
    }
    
    const chunk = Buffer.from(`${this.count++}\n`);
    this.push(chunk); // Push the chunk into the stream buffer
  }
}

const myCounter = new CounterStream(5);
myCounter.on('data', (data) => console.log(data.toString()));

Why this is robust

Because the _read method only executes when the internal buffer has space, you don't overwhelm the consumer. If the consumer is slow (like a slow disk write), the stream naturally pauses because the _read method won't be called again until the buffer clears. This is the foundation of preventing memory overflow in Node.js applications. By controlling the frequency of this.push(), you maintain precise control over the memory footprint of your data source.

Key Takeaway: A custom Readable stream is a state machine that only generates data when requested by the consumer via the _read method, ensuring memory efficiency through demand-based supply.