Course contentsShow
← Python
Lesson 632 of 1,06010. Advanced Language FeaturesFree lesson

Maintaining Infinite Data Streams with Generators

You will construct infinite generators to handle continuous data streams, such as sensor readings or logs, without exhausting system memory.

In this lesson, you will learn how to leverage Python generators to process continuous, never-ending data streams without crashing your system’s memory.

The Infinite Loop Problem

When you process data, you typically use a list to store results. However, if you are monitoring a live sensor or reading a continuous server log, that list will grow until it consumes all available RAM. Generators solve this by producing values on-demand, one at a time, rather than storing them all at once.

The secret is using an infinite while True loop inside a generator function. Because generators use yield to pause execution, the function doesn't actually "finish." It simply waits until the next value is requested.

Implementing a Sensor Stream

Imagine we are monitoring a temperature sensor. We want to simulate a stream that provides a new reading every time we ask for it:

import random

def temperature_sensor():
    while True:
        # Simulate a reading from a hardware sensor
        reading = 20 + random.random() * 5
        yield reading

# Usage
sensor_stream = temperature_sensor()

# We can safely iterate over this "infinite" stream
for _ in range(3):
    print(f"Current Temperature: {next(sensor_stream):.2f}°C")

In the code above, temperature_sensor never returns; it runs indefinitely. When we call next(sensor_stream), the function resumes exactly where it left off, computes the next value, yields it, and pauses again. The memory footprint remains constant regardless of whether we process ten readings or ten billion.

Why this works

Traditional functions are "eager"—they calculate and return everything at once. Generators are "lazy." They maintain their internal state (like local variables and the loop position) but do not hold the history of previously generated values unless you explicitly store them. This makes them the perfect tool for handling real-time data pipelines where you only care about the most recent piece of information.

Key Takeaway: By combining an infinite while True loop with the yield keyword, you create a memory-efficient pipeline that processes continuous data streams on-demand without exhausting system resources.