In this lesson, you will learn how to bridge data communication between independent Node.js processes using stdin and stdout as a high-performance pipe.
The Bridge Between Processes
In a multi-process architecture, we often want to split heavy tasks—like image processing or data parsing—into separate workers to avoid blocking the main event loop. While you could use sockets or message queues, the simplest and most efficient way to link two processes is by connecting the stdout of one to the stdin of the other.
When you spawn a child process in Node.js, you get access to its standard I/O streams. By piping these streams, you create a direct data bridge where the output of the "producer" process flows seamlessly into the input of the "consumer" process.
Implementation Example
Imagine we have a producer.js that generates data and a consumer.js that processes it. We can wire them together using the child_process module:
const { spawn } = require('child_process');
// Spawn the producer and consumer
const producer = spawn('node', ['producer.js']);
const consumer = spawn('node', ['consumer.js']);
// Pipe the output of the producer directly into the input of the consumer
producer.stdout.pipe(consumer.stdin);
// Handle data from the consumer
consumer.stdout.on('data', (data) => {
console.log(`Consumer result: ${data}`);
});
Why this is powerful
This approach leverages the operating system’s underlying pipe buffers. Because pipe() inherently respects backpressure, if the consumer process is slower than the producer, the OS will naturally throttle the producer. This prevents memory bloat because the data is transferred between process memory spaces via system-level buffers rather than loading everything into a single process's heap.
By decoupling the processes through streams, you ensure that your main application remains responsive even while the heavy-lifting sub-process is churning through data. You aren't just sending messages; you are creating a continuous, flow-controlled pipeline that scales naturally across your CPU cores.
Key Takeaway: Cross-process streaming uses stdin and stdout to create a backpressure-aware bridge between processes, allowing them to collaborate on data processing without overloading your main event loop.