Designing Resilient Async Consumers

You will build consumer logic that can recover from crashes while maintaining its place in an async task queue.

In this lesson, you will learn how to design an asynchronous consumer that keeps track of its progress, ensuring that if a crash occurs, the system knows exactly where to resume work.

The Problem: The "At-Least-Once" Dilemma

In an event-driven system, when a consumer pulls a task from a queue, there is a risk that the process crashes after receiving the task but before finishing it. If your consumer simply "forgets" the task upon crashing, that work is lost forever. To build a resilient consumer, we need a way to acknowledge the completion of a task only after it is fully processed, while keeping a persistent record of pending work.

Maintaining Progress

The core pattern for resilience is to split the consumption process into three distinct steps: Fetch, Process, and Acknowledge.

If your consumer fails, the "Acknowledge" step never executes. Because the task was never marked as complete in your persistent storage (or queue provider), the system can safely re-queue or re-assign that task when the process restarts.

Consider this simplified logic pattern:

async function processTask(task) {
  try {
    // 1. Mark task as 'in-progress' in persistent storage
    await db.updateStatus(task.id, 'processing');
    
    // 2. Execute the business logic
    await performWork(task);
    
    // 3. Delete or mark as 'done' ONLY after success
    await db.remove(task.id);
  } catch (err) {
    // If we crash here, the task remains in the DB
    // A restart script can pick up 'processing' items
    console.error('Task failed, state preserved for retry');
  }
}

Why this matters

By treating your local memory state as ephemeral and your external storage as the "source of truth," you transform your consumer from a fragile process into a resilient one. Even if your Node.js event loop terminates abruptly due to an unhandled exception or memory limit, the state of the task is safely locked in your database. Upon reboot, the consumer simply queries for any tasks left in the processing state and resumes execution from that point.

Key Takeaway: Never consider a task finished until your persistent storage confirms the acknowledgment, allowing your consumer to recover its position after any crash.