Course contentsShow
← Cloud and Infrastructure
Lesson 499 of 2,9899. Serverless ComputeFree lesson

Fan-Out/Fan-In Pattern

Distributing work across multiple concurrent function invocations and aggregating results.

Fan-Out/Fan-In Pattern

What you'll learn: How to split a large workload across many concurrent functions and combine their results efficiently.


The Core Idea

The Fan-Out/Fan-In pattern is a serverless architecture technique where you distribute a single large task across multiple concurrent function invocations (fan-out), then aggregate all their results back into a single output (fan-in). Think of it like dividing a big pizza order among several drivers (fan-out), then collecting all the deliveries at one address (fan-in).

This pattern leverages the automatic scaling of serverless functions to process work in parallel, dramatically reducing total processing time compared to sequential execution.

How It Works

Fan-Out: A coordinator function receives a large dataset or task list and splits it into smaller chunks. It then invokes multiple function instances simultaneously, each processing one chunk independently. Since functions scale automatically based on concurrency, hundreds or thousands can run at once.

Fan-In: As each worker function completes, it writes its result to a shared location (like object storage or a database). The coordinator either waits for all workers to finish or monitors completion through another mechanism, then aggregates the results into a final output.

Common Use Cases

  • Processing large datasets (image resizing, log parsing)
  • Parallel API calls to external services
  • Batch data transformations
  • MapReduce-style operations

Important Considerations

You must handle partial failures gracefully—some workers may timeout or error. Use idempotent functions and consider leveraging function retries (already covered) alongside storage-based coordination.

Key Takeaway: The Fan-Out/Fan-In pattern distributes work across many concurrent functions for parallel processing, then aggregates results, turning minutes or hours of sequential work into seconds of parallel execution.