Course contentsShow
← System Design
Lesson 1712 of 1,91941. Real-World System: Notification SystemFree lesson

Deduplication Windows and Storage

Implementing time-bounded deduplication using in-memory caches or databases for recent deliveries.

Deduplication Windows and Storage

What You'll Learn

You'll learn how to implement time-bounded deduplication using in-memory caches or databases to prevent duplicate notification deliveries within a specific timeframe.

The Core Idea

After establishing idempotency keys (lesson 1711), you need a practical storage mechanism to remember which notifications you've already sent. This is where deduplication windows come in: you define a time period during which duplicates are blocked, and choose storage that can efficiently check "have I seen this idempotency key recently?"

Think of it like a bouncer at a club with a guest list that expires after 24 hours. The bouncer doesn't need to remember everyone who visited last year—just today's entries.

Storage Options

In-Memory Caches (Redis, Memcached):

  • Store idempotency keys with TTL (time-to-live)
  • Ultra-fast lookups (microseconds)
  • Keys automatically expire after your deduplication window
  • Example: SET notification:idempotency:abc123 1 EX 86400 (24-hour window)

Databases (PostgreSQL, DynamoDB):

  • More durable but slower
  • Use indexed timestamp columns for efficient queries
  • Require periodic cleanup jobs to delete expired entries
  • Better for audit trails or longer windows

Choosing Your Window

Common deduplication windows:

  • Minutes (5-15): For transient errors and immediate retries
  • Hours (1-24): Standard approach for most notifications
  • Days (7): For critical "never duplicate" scenarios

Longer windows = more storage cost. Balance user experience against infrastructure expense.

Implementation Pattern

  1. On incoming notification: generate or extract idempotency key
  2. Check cache: EXISTS notification:idempotency:{key}
  3. If exists → skip delivery (duplicate)
  4. If not → deliver notification and store key with TTL

Key Takeaway: Deduplication windows let you prevent duplicate notifications by storing idempotency keys in fast, TTL-enabled caches for a bounded time period—balancing reliability with storage efficiency.