Fanout Worker Parallelization
You'll learn how to speed up fanout operations by dividing follower lists into batches and processing them simultaneously across multiple workers.
The Core Idea
When a user posts to social media, the fanout service must write that post to thousands—or millions—of followers' feeds. Processing this sequentially would take too long. Instead, you partition the follower list into batches and assign each batch to a separate worker that runs in parallel.
Think of it like stuffing wedding invitations: one person stuffing 10,000 envelopes would take all day, but ten people each stuffing 1,000 envelopes simultaneously finish in an hour.
How It Works
-
Partition Followers: When a post is created, the fanout service queries the follow graph and splits the follower list into chunks (e.g., batches of 500 or 1,000 follower IDs).
-
Distribute to Workers: Each batch is sent to a different worker instance—typically via the message queue you set up for asynchronous fanout processing.
-
Parallel Execution: Workers independently fetch each batch, then write the post reference to each follower's feed cache or storage.
-
Aggregation: No explicit "join" is needed—each worker completes independently, and the fanout is done when all workers finish.
Tuning Batch Size
- Too small: Overhead of coordinating many tasks
- Too large: One slow worker becomes a bottleneck
- Sweet spot: Usually 500–2,000 followers per batch, depending on your infrastructure
This approach scales horizontally: more workers = faster fanout, especially critical for high-throughput systems.
Key Takeaway:
Fanout worker parallelization splits a user's follower list into batches and processes them concurrently across multiple workers, dramatically reducing fanout latency and enabling horizontal scalability.