Deduplication Windows and Storage
What You'll Learn
You'll learn how to implement time-bounded deduplication using in-memory caches or databases to prevent duplicate notification deliveries within a specific timeframe.
The Core Idea
After establishing idempotency keys (lesson 1711), you need a practical storage mechanism to remember which notifications you've already sent. This is where deduplication windows come in: you define a time period during which duplicates are blocked, and choose storage that can efficiently check "have I seen this idempotency key recently?"
Think of it like a bouncer at a club with a guest list that expires after 24 hours. The bouncer doesn't need to remember everyone who visited last year—just today's entries.
Storage Options
In-Memory Caches (Redis, Memcached):
- Store idempotency keys with TTL (time-to-live)
- Ultra-fast lookups (microseconds)
- Keys automatically expire after your deduplication window
- Example:
SET notification:idempotency:abc123 1 EX 86400(24-hour window)
Databases (PostgreSQL, DynamoDB):
- More durable but slower
- Use indexed timestamp columns for efficient queries
- Require periodic cleanup jobs to delete expired entries
- Better for audit trails or longer windows
Choosing Your Window
Common deduplication windows:
- Minutes (5-15): For transient errors and immediate retries
- Hours (1-24): Standard approach for most notifications
- Days (7): For critical "never duplicate" scenarios
Longer windows = more storage cost. Balance user experience against infrastructure expense.
Implementation Pattern
- On incoming notification: generate or extract idempotency key
- Check cache:
EXISTS notification:idempotency:{key} - If exists → skip delivery (duplicate)
- If not → deliver notification and store key with TTL
Key Takeaway: Deduplication windows let you prevent duplicate notifications by storing idempotency keys in fast, TTL-enabled caches for a bounded time period—balancing reliability with storage efficiency.