In this lesson, you will learn how to connect multiple generator functions into a sequence to create modular, memory-efficient data processing pipelines.
Building a Sequential Pipeline
When processing large datasets, you rarely perform just one action. You might need to filter data, then transform it, and finally format it. Instead of loading everything into memory, you can link generator functions together so that data flows through them one piece at a time, like water through a series of pipes.
Because generators use the yield statement to produce values lazily, the next generator in your chain only executes when the previous one requests a new item. This means your entire pipeline holds only a single "item" in memory at any given moment.
How to Chain Them
To chain generators, you simply pass the output of one generator function as the input to the next.
def get_numbers():
for i in range(10):
yield i
def square_numbers(source):
for num in source:
yield num ** 2
def filter_even(source):
for num in source:
if num % 2 == 0:
yield num
# Linking the pipeline
numbers = get_numbers()
squared = square_numbers(numbers)
even_squares = filter_even(squared)
for val in even_squares:
print(val)
In this example, get_numbers creates a stream of integers. square_numbers consumes that stream and yields squares. Finally, filter_even consumes the squared values and passes along only the even results. Notice how the logic is separated: each function is modular, testable, and only cares about its specific task.
Why It Matters
This pattern keeps your code clean by separating concerns. If you decide you need to add a new step—like converting the numbers to strings—you simply create a new generator function and insert it into the chain. You never have to create large intermediate lists, keeping your application fast and lightweight even when processing millions of data points.
Key Takeaway: By passing one generator as an argument to another, you create a memory-efficient pipeline where data is processed lazily and sequentially through modular, reusable stages.