Course contentsShow
← Python
Lesson 316 of 1,0605. Advanced Iteration and ComprehensionsFree lesson

Advanced Filtering and Mapping with Generator Pipes

You will create reusable functional components that filter and map data streams without storing intermediate results in memory.

In this lesson, you will learn how to build modular, memory-efficient data processing pipelines by chaining specialized generator functions that filter and map data on the fly.

The Power of "Streaming" Logic

When we process large datasets, loading everything into a list can crash your memory. Instead of performing one giant transformation, we can break our logic into discrete, reusable steps: filtering (selecting specific items) and mapping (transforming items). By linking these as generators, each item "flows" through the entire pipeline one-by-one, requiring only enough memory to hold the current object being processed.

Building the Pipeline

Imagine you have a massive stream of log entries and you only want to process the error messages by converting them to uppercase. We can treat these steps as independent functions:

def get_errors(stream):
    for entry in stream:
        if "ERROR" in entry:
            yield entry

def to_uppercase(stream):
    for entry in stream:
        yield entry.upper()

# The raw data stream (e.g., from a file)
log_data = ["INFO: start", "ERROR: disk full", "INFO: retry", "ERROR: timeout"]

# Compose the pipeline
error_stream = get_errors(log_data)
final_stream = to_uppercase(error_stream)

# Execution happens here, lazily
for processed_line in final_stream:
    print(processed_line)

Why This Matters

By isolating logic into get_errors and to_uppercase, your code becomes highly modular. You can easily add a third step—like saving to a database or calculating a timestamp—without ever modifying the original functions or inflating your memory usage.

Because we use yield instead of returning a list, Python never calculates the "middle" state of the data. The first function produces one item, the second function consumes it immediately, and the result is printed. Once that item is processed, the memory is cleared, and the next item begins its journey through the pipe. This approach turns complex data transformations into clean, readable, and highly scalable streams.

Key Takeaway: By decoupling filtering and mapping into separate generator functions, you create modular, memory-efficient pipelines that process data one piece at a time instead of storing entire collections in RAM.