Course contentsShow
← System Design
Lesson 1035 of 1,91926. Idempotency and RetriesFree lesson

Idempotency in Event Processing

Tracking processed event IDs and using deduplication windows to ensure event handlers run exactly once.

Idempotency in Event Processing

What You'll Learn

You'll learn how to ensure event handlers process each event exactly once, even when events are delivered multiple times.

The Challenge in Event-Driven Systems

In distributed event processing systems (like Kafka, RabbitMQ, or AWS EventBridge), at-least-once delivery is the common guarantee. This means the same event might arrive multiple times due to retries, network issues, or consumer restarts. Without safeguards, your handler might process a payment twice, send duplicate emails, or corrupt data.

Tracking Processed Event IDs

The solution mirrors the idempotency key pattern you've already learned. Each event should carry a unique event ID (often a UUID or combination of source + timestamp). Your event handler:

  1. Checks if this event ID has been processed before
  2. Skips processing if it's a duplicate
  3. Records the event ID after successful processing

Think of it like a bouncer at a club with a guest list. Even if someone tries to enter twice with the same ticket, the bouncer checks the list and only lets them in once.

Deduplication Windows

You can't store event IDs forever—storage would grow infinitely. Instead, use a deduplication window: track event IDs for a fixed time period (e.g., 7 days). Events older than this window are assumed fresh, but in practice, duplicate deliveries happen within minutes or hours, not days.

The window acts like a rolling checkpoint, balancing memory efficiency with practical duplicate detection.

Implementation Pattern

Store processed event IDs in a fast lookup system (Redis, DynamoDB, or a database table with the event ID as primary key). Before processing an event, check this store. After successful processing, insert the event ID with an expiration timestamp.

Key Takeaway: Event processing idempotency requires tracking processed event IDs in a persistent store with a reasonable deduplication window, ensuring handlers execute exactly once despite at-least-once delivery guarantees.