In this lesson, you will learn how to build efficient, modular data processing pipelines by linking multiple coroutines together to transform data in stages.
Building Data Pipelines with Coroutines
In a traditional approach, processing a dataset often involves creating large intermediate lists that consume significant memory. By using coroutines, we can treat our data processing steps as a sequence of "processing stations" that handle items one at a time as they arrive.
A coroutine-based pipeline consists of three main roles:
- The Source: A generator that produces raw data.
- The Filters/Transformers: Coroutines that receive data, perform a transformation, and
send()it to the next step. - The Sink: The final coroutine that consumes the processed data (e.g., printing or writing to a file).
How it Works
To build a pipeline, each stage must be "primed" by calling next() on it. This advances the execution to the first yield statement, allowing the coroutine to begin receiving data.
Consider a simple pipeline that filters numbers and squares them:
def filter_evens(target):
while True:
item = yield
if item % 2 == 0:
target.send(item)
def square_numbers(target):
while True:
item = yield
target.send(item ** 2)
def printer():
while True:
item = yield
print(f"Processed result: {item}")
# Setup the pipeline
sink = printer()
next(sink)
transformer = square_numbers(sink)
next(transformer)
pipeline = filter_evens(transformer)
next(pipeline)
# Send data into the start of the pipeline
for i in range(10):
pipeline.send(i)
In this architecture, pipeline acts as the entry point. When we send() a value to it, the filter_evens coroutine decides whether to forward the value. If it does, square_numbers receives it, performs the calculation, and passes it to the printer sink.
Because we use yield and send(), data flows through the entire pipeline without ever needing to store the full dataset in memory at once. This makes your application highly modular—you can swap, remove, or reorder pipeline stages simply by changing the targets passed to the coroutines.
Key Takeaway: Coroutine pipelines transform data in a streaming fashion by priming stages and chaining them together, allowing for memory-efficient, decoupled data processing.