Detecting and Eliminating Message Duplication

In this lesson, you will learn how to implement deduplication buffers to ensure that your background tasks are processed exactly once, even when your message broker guarantees only "at-least-once" delivery.

The Problem: At-Least-Once Reality

In robust distributed systems, network hiccups often force brokers to redeliver messages. If your worker crashes after completing a task but before acknowledging it to the queue, the broker will send that task again. Without deduplication, your system might process the same job twice—leading to double-charged payments or duplicate database entries.

The Solution: The Deduplication Buffer

To solve this, we use an atomic Deduplication Buffer, typically implemented with Redis. Before processing a message, the worker checks if the task's unique ID exists in the buffer.

Here is how you can implement this in your Node.js worker:

async function processTask(task) {
  const isDuplicate = await redis.set(
    `task_id:${task.id}`, 
    'processing', 
    'NX', 
    'EX', 60 // Set if not exists, expire in 60s
  );

  if (!isDuplicate) {
    console.log(`Duplicate detected: ${task.id}. Skipping.`);
    return;
  }

  try {
    await performWork(task);
  } catch (err) {
    // If work fails, delete the key so it can be retried
    await redis.del(`task_id:${task.id}`);
    throw err;
  }
}

How it Works

1. Atomic Check-and-Set: By using SET key value NX, we ensure that only one worker can "claim" the task ID. If the key exists, the operation returns null, signaling that another worker is currently handling or has already handled the task.
2. TTL (Time-to-Live): The EX parameter ensures that if a worker crashes, the lock will eventually expire, allowing the task to be safely retried later.
3. Idempotency: This buffer acts as a guardrail. Even if your message broker sends the same message five times, only the first request will pass the isDuplicate check, ensuring your business logic runs exactly once.

Key Takeaway: Use atomic Redis operations to create a temporal lock for each message ID, effectively shielding your downstream logic from the side effects of redundant at-least-once delivery.