Course contentsShow
← Python
Lesson 634 of 1,06010. Advanced Language FeaturesFree lesson

Implementing Pipeline Architectures with Coroutines

You will build modular data processing pipelines where multiple coroutines pass data between each other to perform complex multi-stage transformations.

In this lesson, you will learn how to build efficient, modular data processing pipelines by linking multiple coroutines together to transform data in stages.

Building Data Pipelines with Coroutines

In a traditional approach, processing a dataset often involves creating large intermediate lists that consume significant memory. By using coroutines, we can treat our data processing steps as a sequence of "processing stations" that handle items one at a time as they arrive.

A coroutine-based pipeline consists of three main roles:

  1. The Source: A generator that produces raw data.
  2. The Filters/Transformers: Coroutines that receive data, perform a transformation, and send() it to the next step.
  3. The Sink: The final coroutine that consumes the processed data (e.g., printing or writing to a file).

How it Works

To build a pipeline, each stage must be "primed" by calling next() on it. This advances the execution to the first yield statement, allowing the coroutine to begin receiving data.

Consider a simple pipeline that filters numbers and squares them:

def filter_evens(target):
    while True:
        item = yield
        if item % 2 == 0:
            target.send(item)

def square_numbers(target):
    while True:
        item = yield
        target.send(item ** 2)

def printer():
    while True:
        item = yield
        print(f"Processed result: {item}")

# Setup the pipeline
sink = printer()
next(sink)

transformer = square_numbers(sink)
next(transformer)

pipeline = filter_evens(transformer)
next(pipeline)

# Send data into the start of the pipeline
for i in range(10):
    pipeline.send(i)

In this architecture, pipeline acts as the entry point. When we send() a value to it, the filter_evens coroutine decides whether to forward the value. If it does, square_numbers receives it, performs the calculation, and passes it to the printer sink.

Because we use yield and send(), data flows through the entire pipeline without ever needing to store the full dataset in memory at once. This makes your application highly modular—you can swap, remove, or reorder pipeline stages simply by changing the targets passed to the coroutines.

Key Takeaway: Coroutine pipelines transform data in a streaming fashion by priming stages and chaining them together, allowing for memory-efficient, decoupled data processing.