Course contentsShow
← System Design
Lesson 686 of 1,91918. Message Queues and BrokersFree lesson

Idempotency and Deduplication

Building idempotent consumers and using deduplication IDs to handle duplicate deliveries safely.

Idempotency and Deduplication

What you'll learn: How to handle duplicate message deliveries safely using idempotency and deduplication IDs.

The Problem with At-Least-Once Delivery

When using at-least-once delivery guarantees (which we covered earlier), messages can be delivered more than once. A network hiccup might cause a consumer to process a message, crash before acknowledging it, and then receive the same message again. If your consumer charges a credit card or decrements inventory, processing twice creates serious problems.

What Is Idempotency?

Idempotency means an operation can be applied multiple times without changing the result beyond the first application. Think of a light switch: flipping it "on" three times has the same effect as flipping it once—the light is on.

An idempotent consumer is designed so that processing the same message multiple times produces the same outcome as processing it once.

Building Idempotent Consumers

The key technique is using deduplication IDs (also called idempotency keys). Each message carries a unique identifier. The consumer tracks which IDs it has already processed:

  1. Consumer receives message with ID msg-12345
  2. Check: "Have I processed msg-12345 before?"
  3. If no: process the message and store msg-12345 in a processed-IDs database
  4. If yes: skip processing, just acknowledge the message

Storage Considerations

Store deduplication IDs in a database (SQL, Redis, DynamoDB) with appropriate retention. You don't need to keep IDs forever—typically hours or days depending on your redelivery window.

Natural Idempotency

Sometimes operations are naturally idempotent: "Set user status to active" can run multiple times safely. When possible, design your business logic this way first.

Key Takeaway: Idempotent consumers with deduplication IDs prevent duplicate processing in at-least-once delivery systems, ensuring operations like payments or inventory updates happen exactly once from a business perspective, even if messages arrive multiple times.