Course contentsShow
← Python
Lesson 639 of 1,06010. Advanced Language FeaturesFree lesson

Benchmarking Memory Efficiency in Iteration Patterns

You will profile your code to quantify the memory savings achieved by replacing list-based processing with lazy generator-based iteration techniques.

In this lesson, you will learn how to quantify the memory footprint differences between eager list-based processing and lazy generator-based iteration using the tracemalloc module.

The Cost of Eager Loading

When you process data using list comprehensions or standard lists, Python must allocate enough contiguous memory to store every element of the collection simultaneously. If your dataset contains millions of items, this can trigger a MemoryError or force your system to swap to disk, drastically slowing down performance.

Generators, conversely, generate values on-the-fly. They only track the current state, requiring a constant, minimal amount of memory regardless of the dataset size. To prove this efficiency, we use the standard library’s tracemalloc module to capture "peak" memory usage during execution.

Profiling the Difference

Consider a scenario where we process a range of 10 million integers. We can compare a list comprehension against a generator expression:

import tracemalloc

def profile_memory(func):
    tracemalloc.start()
    func()
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return peak / 10**6  # Return peak in MB

# Eager: Creates a list of 10 million items in memory
def eager_approach():
    data = [x * 2 for x in range(10_000_000)]
    return sum(data)

# Lazy: Evaluates one item at a time
def lazy_approach():
    data = (x * 2 for x in range(10_000_000))
    return sum(data)

print(f"List peak: {profile_memory(eager_approach):.2f} MB")
print(f"Generator peak: {profile_memory(lazy_approach):.2f} MB")

When you run this, you will see the list version consume significantly more memory (often over 80MB for integers) compared to the generator, which will likely show a peak usage of less than 1MB. tracemalloc highlights the "high-water mark" of memory allocation, providing empirical proof that lazy evaluation avoids the massive overhead of keeping entire datasets in RAM. By benchmarking, you transition from guessing about efficiency to confirming that your iteration patterns are scaling sustainably.

Key Takeaway: Use tracemalloc to capture the peak memory allocation of your iterables, confirming that generators maintain a constant memory footprint while list-based approaches scale linearly with dataset size.