Course contentsShow
← System Design
Lesson 1656 of 1,91940. Real-World System: Social Media FeedFree lesson

Fanout Failure Handling

Implementing retries, dead letter queues, and partial success handling when fanout to some followers fails.

Fanout Failure Handling

What you'll learn: How to gracefully handle failures when delivering posts to follower feeds during the fanout process, ensuring reliability at massive scale.

The Challenge

When fanning out a post to millions of followers, network issues, database timeouts, or service outages will inevitably cause some deliveries to fail. You can't simply give up—users must eventually see the post.

Core Strategies

Retries with Exponential Backoff

When a fanout worker fails to write to a follower's feed cache, retry the operation with increasing delays (1s, 2s, 4s, etc.). This gives temporary issues time to resolve without overwhelming already-struggling systems.

Dead Letter Queues (DLQ)

After exhausting retries (e.g., 3-5 attempts), move failed fanout tasks to a dead letter queue—a separate storage for "poison messages." These can be manually investigated, reprocessed during off-peak hours, or trigger alerts for engineering teams.

Think of it like package delivery: if your address is temporarily inaccessible, the courier tries again tomorrow. If it still fails after several attempts, the package goes to a depot for special handling.

Partial Success Handling

Fanout to 10 million followers shouldn't be all-or-nothing. Track progress granularly (which followers received the post) so you can resume from where you left off rather than restarting. Mark the fanout job as "partially complete" and continue processing remaining followers.

Idempotency

Ensure fanout operations are idempotent—writing the same post ID to a follower's feed twice doesn't create duplicates. This makes retries safe and prevents showing users the same post multiple times.

Monitoring and Alerting

Track fanout failure rates, DLQ depth, and retry patterns. Spike in failures? You need immediate visibility to address systemic issues before they cascade.

Key Takeaway: Robust fanout systems combine automatic retries with exponential backoff, dead letter queues for persistent failures, partial success tracking for resumability, and idempotent operations to make retries safe—ensuring eventual consistency even when infrastructure hiccups.