Designing Idempotent Job Handlers

Implement logic that ensures tasks can be safely executed multiple times without adverse side effects.

Designing Idempotent Job Handlers

In this lesson, you will learn how to design job handlers that guarantee the same outcome regardless of how many times a task is executed.

The Problem of "At-Least-Once" Delivery

In distributed task queues, network failures or process crashes often force the system to retry jobs. If a job involves charging a credit card or sending an email, retrying that job blindly could result in double charges or duplicate notifications. This is the danger of "at-least-once" delivery patterns; your code must be smart enough to recognize when it has already finished its work.

The Idempotency Key Pattern

The most robust way to handle this is by introducing an Idempotency Key. Before executing any side-effect-heavy logic, your handler should check a persistent store (like Redis or your database) to see if that specific task ID has already been marked as COMPLETED.

Consider this implementation pattern:

async function processPayment(job) {
  const { jobId, amount, userId } = job.data;

  // 1. Check if we've already processed this exact job
  const existingResult = await db.processedJobs.findOne({ jobId });
  if (existingResult) {
    return existingResult.response;
  }

  // 2. Perform the side effect (e.g., call payment API)
  const paymentResponse = await stripe.charges.create({ amount, userId });

  // 3. Mark as complete atomically
  await db.processedJobs.insertOne({
    jobId,
    status: 'COMPLETED',
    response: paymentResponse,
    timestamp: new Date()
  });

  return paymentResponse;
}

Why This Works

By separating the "check for existence" and "mark as complete" steps, you create a safeguard. Even if the Node.js process crashes exactly after the payment API call but before the database update, the next retry will see the state is not yet COMPLETED and proceed.

If the database update happens first but the network blips, the second attempt will see the COMPLETED flag and exit early, preventing the payment API from being called a second time. This logic turns a fragile operation into a resilient, repeatable process.

Key Takeaway: An idempotent handler uses a unique identifier to verify the state of a task before executing side effects, ensuring that duplicate processing never leads to duplicate results.