Flow Control and Backpressure

In this lesson, you will learn how to design stream pipelines that automatically signal when they are overwhelmed, preventing memory overflow during high-speed data ingestion.

The Problem: The "Firehose" Effect

Imagine your Node.js application is a water pipe. You are reading a massive 10GB file from disk (the source) and sending it over a network socket (the destination). If the disk reads data faster than the network can transmit it, Node.js will store that excess data in an internal buffer. If the network is consistently slower, that buffer grows infinitely until your process crashes with an Out of Memory error. This is a buffer overflow.

How Backpressure Works

Backpressure is the signaling mechanism that tells the source to "slow down" when the destination buffer is full.

When you use the .pipe() method in Node.js, the framework handles this logic for you automatically. When the destination stream’s buffer reaches its "high water mark" (the maximum allowed size), it returns false to the source stream. The source stream then pauses its data flow. Once the destination finishes clearing its buffer, it emits a 'drain' event, signaling the source to resume reading.

A Manual Implementation

If you aren't using .pipe(), you must respect backpressure manually using the return value of write():

// A simple pattern to respect backpressure
function writeData(source, destination) {
  source.on('data', (chunk) => {
    const canWrite = destination.write(chunk);
    
    // If the buffer is full, pause the source
    if (!canWrite) {
      source.pause();
      
      // Resume when the destination is ready again
      destination.once('drain', () => {
        source.resume();
      });
    }
  });
}

By checking the boolean return value of destination.write(), you ensure that you aren't pumping data into memory faster than the network can handle. This creates a balanced, stable pipeline where the data flow is throttled by the slowest component in the chain, effectively protecting your application from crashing under heavy load.

Key Takeaway: Backpressure is the feedback loop that prevents memory exhaustion by signaling a source to pause whenever the destination’s buffer is saturated.