In this lesson, you will learn how to build modular, memory-efficient data pipelines by composing multiple generator functions to transform streams of data in a single pass.
The Power of Composition
When processing large datasets, you rarely perform just one operation. You might need to filter raw data, transform it, and then format it. Instead of creating massive intermediate lists—which consume significant RAM—you can "chain" generator functions together. This creates a pipeline where data flows through each stage one item at a time.
Imagine a pipeline that processes a stream of numbers: filtering out odd ones, squaring the survivors, and converting them into strings. Instead of doing this in one monolithic block, we define small, focused generators:
def get_even_numbers(numbers):
for n in numbers:
if n % 2 == 0:
yield n
def square_numbers(numbers):
for n in numbers:
yield n ** 2
def format_as_string(numbers):
for n in numbers:
yield f"Result: {n}"
# Composing the pipeline
data = range(10)
pipeline = format_as_string(square_numbers(get_even_numbers(data)))
# Data flows only when we iterate
for item in pipeline:
print(item)
Why Compose Generators?
By composing these functions, we achieve lazy evaluation. Notice that get_even_numbers feeds directly into square_numbers, which feeds into format_as_string. No calculations occur until the for loop requests the next value.
This approach offers two major benefits:
- Memory Efficiency: Only one number exists in memory at any given stage of the pipeline at any time.
- Readability & Reuse: Each generator is a small, testable unit. You can swap the order of operations, add new stages, or remove them without rewriting your entire logic.
By treating data as a flowing stream rather than a static container, you transform complex, multi-step processing into a clean, declarative sequence of operations that scales to datasets of any size.
Key Takeaway: Generator composition allows you to build sophisticated, memory-efficient data pipelines by chaining modular processing steps that transform data lazily, one element at a time.