In this lesson, you will learn how to leverage Python generators to process continuous, never-ending data streams without crashing your system’s memory.
The Infinite Loop Problem
When you process data, you typically use a list to store results. However, if you are monitoring a live sensor or reading a continuous server log, that list will grow until it consumes all available RAM. Generators solve this by producing values on-demand, one at a time, rather than storing them all at once.
The secret is using an infinite while True loop inside a generator function. Because generators use yield to pause execution, the function doesn't actually "finish." It simply waits until the next value is requested.
Implementing a Sensor Stream
Imagine we are monitoring a temperature sensor. We want to simulate a stream that provides a new reading every time we ask for it:
import random
def temperature_sensor():
while True:
# Simulate a reading from a hardware sensor
reading = 20 + random.random() * 5
yield reading
# Usage
sensor_stream = temperature_sensor()
# We can safely iterate over this "infinite" stream
for _ in range(3):
print(f"Current Temperature: {next(sensor_stream):.2f}°C")
In the code above, temperature_sensor never returns; it runs indefinitely. When we call next(sensor_stream), the function resumes exactly where it left off, computes the next value, yields it, and pauses again. The memory footprint remains constant regardless of whether we process ten readings or ten billion.
Why this works
Traditional functions are "eager"—they calculate and return everything at once. Generators are "lazy." They maintain their internal state (like local variables and the loop position) but do not hold the history of previously generated values unless you explicitly store them. This makes them the perfect tool for handling real-time data pipelines where you only care about the most recent piece of information.
Key Takeaway: By combining an infinite while True loop with the yield keyword, you create a memory-efficient pipeline that processes continuous data streams on-demand without exhausting system resources.