Building Resilient HTTP Streaming

In this lesson, you will learn how to maintain stable HTTP connections that stream data incrementally to clients even when network conditions become unpredictable.

When you stream data over HTTP—like a long-lived log stream or a large data export—the connection is fragile. If the client’s network slows down or hiccups, the server might try to push data faster than the client can receive it. In Node.js, if you ignore these fluctuations, you risk crashing your process due to memory bloat or causing the TCP connection to hang.

To build a resilient stream, you must bridge the gap between your data source and the http.ServerResponse object. The secret is to respect the drain event and the write() method's return value.

Implementing Backpressure in HTTP

When you call response.write(), it returns a boolean. If it returns false, it means the internal buffer is full and the client is struggling to keep up. You must pause your data source immediately to prevent memory leaks and wait for the drain event before resuming.

const http = require('http');

http.createServer((req, res) => {
  const source = getMyDataStream(); // A hypothetical readable stream

  source.on('data', (chunk) => {
    const canContinue = res.write(chunk);
    
    // If the buffer is full, pause the source stream
    if (!canContinue) {
      source.pause();
      
      // Resume when the buffer is clear and the client is ready
      res.once('drain', () => {
        source.resume();
      });
    }
  });

  source.on('end', () => res.end());
}).listen(3000);

By manually orchestrating pause(), resume(), and the drain event, you ensure that your server only pushes data when the network pipe can actually accommodate it. This approach transforms a fragile HTTP request into a robust, self-regulating pipeline. Even if the user experiences intermittent Wi-Fi or a slow mobile connection, your server will wait patiently, holding the data in a controlled manner rather than overwhelming the socket.

Key Takeaway: A resilient HTTP stream must monitor the return value of res.write() and pause the source stream whenever the buffer is full to prevent memory overflow and connection timeouts.