Advanced Error Recovery

In this lesson, you will learn how to design resilient stream pipelines that automatically recover from stalls or crashes without losing a single byte of in-flight data.

The "Drain and Resume" Challenge

When a pipeline crashes—perhaps due to a temporary network blip or a remote service timeout—the standard behavior is to destroy the stream. However, simply restarting the process often loses the data currently sitting in internal buffers. To achieve robust error recovery, you must treat your pipeline as a stateful sequence that can pick up where it left off.

Designing for Continuity

To avoid data loss, we implement a "checkpoint" mechanism. Instead of pushing raw chunks, we track the progress of our data using an offset. When an error occurs, the pipeline shuts down, but the controller remembers the last successfully processed chunk.

Consider a file-processing pipeline:

let offset = 0;

async function runPipeline() {
  const source = createReadStream('data.log', { start: offset });
  const processor = new TransformStream(/* ... */);

  try {
    await pipeline(source, processor, destination);
  } catch (err) {
    console.error('Pipeline stalled. Restarting from offset:', offset);
    // In a real scenario, update 'offset' based on processor metadata
    setTimeout(runPipeline, 1000); 
  }
}

Preventing Data Loss

The secret to "advanced" recovery is ensuring your Transform streams are idempotent or checkpoint-aware. If your stream is processing a large batch of records, you should emit an event after every successful transformation that updates your global offset.

When the error handler triggers, your ReadStream doesn’t start from byte zero; it starts from the last confirmed successful index. By combining this with the pipeline utility (which manages backpressure automatically), you ensure that when you restart, the system isn't overwhelmed by re-processing already acknowledged data. If the stream stalls due to backpressure, we don't throw an error; we pause, wait for the drain event, and resume, keeping our state intact throughout the entire lifecycle.

Key Takeaway: Never assume a pipeline will finish on the first attempt; build your logic to track offsets and resume from the last successful checkpoint to ensure zero-loss recovery.