Fan-Out/Fan-In Pattern
What you'll learn: How to split a large workload across many concurrent functions and combine their results efficiently.
The Core Idea
The Fan-Out/Fan-In pattern is a serverless architecture technique where you distribute a single large task across multiple concurrent function invocations (fan-out), then aggregate all their results back into a single output (fan-in). Think of it like dividing a big pizza order among several drivers (fan-out), then collecting all the deliveries at one address (fan-in).
This pattern leverages the automatic scaling of serverless functions to process work in parallel, dramatically reducing total processing time compared to sequential execution.
How It Works
Fan-Out: A coordinator function receives a large dataset or task list and splits it into smaller chunks. It then invokes multiple function instances simultaneously, each processing one chunk independently. Since functions scale automatically based on concurrency, hundreds or thousands can run at once.
Fan-In: As each worker function completes, it writes its result to a shared location (like object storage or a database). The coordinator either waits for all workers to finish or monitors completion through another mechanism, then aggregates the results into a final output.
Common Use Cases
- Processing large datasets (image resizing, log parsing)
- Parallel API calls to external services
- Batch data transformations
- MapReduce-style operations
Important Considerations
You must handle partial failures gracefully—some workers may timeout or error. Use idempotent functions and consider leveraging function retries (already covered) alongside storage-based coordination.
Key Takeaway: The Fan-Out/Fan-In pattern distributes work across many concurrent functions for parallel processing, then aggregates results, turning minutes or hours of sequential work into seconds of parallel execution.