Dynamic Stream Routing

In this lesson, you will learn how to build a smart "router" that inspects incoming data chunks in real-time and directs them to specific destinations based on their content.

The Concept of Dynamic Routing

In previous lessons, we used pipelines to move data from a single source to a single destination. However, real-world systems often require us to split a single stream of data based on its content. Imagine a log processor that sends "ERROR" messages to a database and "INFO" messages to a standard file.

Instead of routing data at the end of a process, we use a Transform stream as a router. This stream inspects every chunk that passes through it, decides where it belongs, and sends it to the appropriate destination stream.

Implementing the Router

To route data, your Transform stream needs a reference to your destination streams. You check the chunk’s content, and if it matches your criteria, you write that chunk directly to the destination.

const { Transform } = require('stream');

const createRouter = (errorStream, infoStream) => {
  return new Transform({
    transform(chunk, encoding, callback) {
      const data = chunk.toString();
      
      if (data.includes('ERROR')) {
        errorStream.write(chunk);
      } else {
        infoStream.write(chunk);
      }
      
      callback();
    }
  });
};

Managing Backpressure in Routing

When you manually call destination.write(chunk) inside a router, you must respect backpressure. If the destination stream is full, write() will return false. In a robust system, you should monitor the return value of destination.write() and, if it returns false, pause the readable stream (using source.pause()) and wait for the drain event before resuming.

This pattern transforms a simple linear pipeline into a powerful dispatching system. By embedding the routing logic directly within a Transform stream, you can filter and categorize massive datasets on the fly without ever loading the entire file into memory.

Key Takeaway: Dynamic stream routing allows you to inspect and divert data chunks to different targets mid-flow, provided you manually manage the backpressure of each output destination.