Course contentsShow
← Python
Lesson 630 of 1,06010. Advanced Language FeaturesFree lesson

Processing Large Datasets with Generator Expressions

You will write memory-efficient one-liner generator expressions to perform data transformations on streams without loading entire datasets into memory.

In this lesson, you will learn how to use generator expressions to transform massive datasets on-the-fly without exhausting your system's RAM.

The Memory Bottleneck

When you work with large files—such as multi-gigabyte CSVs or continuous log streams—creating a list of processed results using a list comprehension ([x * 2 for x in data]) forces Python to load the entire collection into memory at once. If the dataset exceeds your available RAM, your program will crash or hang.

The Generator Expression Solution

Generator expressions are the memory-efficient cousin of list comprehensions. Instead of calculating all results upfront and storing them in memory, a generator expression returns an iterator. It yields one item at a time, only when you explicitly ask for the "next" value.

The syntax is identical to a list comprehension, but it uses parentheses () instead of square brackets [].

Practical Example: Filtering and Transforming

Imagine you have a file with millions of sensor readings and you only want to process the "critical" ones (greater than 100) and scale them down:

# Assume sensor_data is a large iterator from a file
raw_data = (line.strip() for line in open("sensors.log"))

# Use a generator expression to process data lazily
processed_data = (int(val) * 0.5 for val in raw_data if int(val) > 100)

# Memory is only used for one item at a time during iteration
for reading in processed_data:
    print(f"Scaled critical reading: {reading}")

In this code, processed_data does not hold any data in memory. It is a "recipe" for data. The int(val) conversion and the multiplication logic only run when the for loop requests the next iteration. If you were to chain five different generator expressions together, Python would still only keep one singular object in memory at any given moment, regardless of how large the underlying file is.

This allows you to build complex "data pipelines" that remain performant regardless of input size, as you are essentially streaming the data through your logic rather than storing it.

Key Takeaway: Use parentheses to create generator expressions when dealing with large datasets to ensure your program processes data one element at a time, keeping your memory footprint constant and minimal.