In this lesson, you will learn how to use the "fan-out" pattern to distribute a single stream of data to multiple destination sinks simultaneously.
The Fan-Out Pattern
When building data pipelines, you often reach a point where a single source—like a live log file or a WebSocket feed—needs to be sent to different destinations. For example, you might want to save incoming data to a database while simultaneously forwarding it to an analytics service.
In Node.js, you cannot simply pipe one Readable stream into multiple Writable streams using .pipe(a).pipe(b), because .pipe() returns the destination stream, not the source. Instead, you use the stream.Readable.prototype.pipe() method to attach multiple destinations to the same source.
Implementation
When you attach multiple destinations, the source stream will broadcast its incoming chunks to every attached sink. However, you must be careful: if one sink experiences backpressure (it’s slower than the source), the source stream will wait for that specific sink to drain before continuing, potentially slowing down all other connected sinks.
const { createReadStream } = require('fs');
const source = createReadStream('data.log');
const sinkA = createWriteStream('backup.log');
const sinkB = createWriteStream('analytics.log');
// Fan-out: Attach both sinks to the same source
source.pipe(sinkA);
source.pipe(sinkB);
Managing the Flow
Because the source stream is now responsible for multiple listeners, keep in mind that the "slowest" sink determines the speed of the entire fan-out operation. If sinkA is a local file (fast) and sinkB is an external API (slow), the network latency of sinkB will force the source to buffer data, which could lead to memory pressure if not managed properly.
In production, consider wrapping your sinks in a PassThrough stream to decouple them. This allows each sink to manage its own backpressure independently, ensuring that a slow downstream consumer doesn't inadvertently crash your main application source.
Key Takeaway: The fan-out pattern allows a single stream to broadcast to multiple sinks, but you must account for the fact that the entire pipeline is only as fast as its slowest consumer.