In this lesson, you will learn how to efficiently merge multiple collections into a single, continuous stream of data using itertools.chain.
The Problem with Intermediate Lists
When you have several distinct collections—such as a list of names, a set of identifiers, and a tuple of configuration settings—you might be tempted to combine them using the + operator: all_data = list1 + list2 + list3. While this works, it forces Python to create a brand-new list in memory, copying every element from your existing collections. If those collections are large, this approach creates unnecessary memory overhead.
The itertools.chain Solution
Instead of creating intermediate objects, itertools.chain treats multiple iterables as a single, unified sequence. It returns an iterator that yields the first element from the first iterable, moves to the next, and continues until all provided iterables are exhausted. Crucially, it does this "lazily," meaning it only keeps track of which iterable it is currently reading, requiring virtually no extra memory.
Consider this example:
from itertools import chain
users = ["Alice", "Bob"]
admins = ("Charlie", "Dana")
guests = {"Eve", "Frank"}
# Combine them into one stream without copying
all_attendees = chain(users, admins, guests)
for person in all_attendees:
print(person)
Flattening Nested Structures
itertools.chain is also highly effective for flattening nested structures. If you have a list of lists and want to process every individual item without manually writing nested for loops, you can use the from_iterable class method:
nested_data = [[1, 2], [3, 4], [5, 6]]
flat_stream = chain.from_iterable(nested_data)
# This will print 1, 2, 3, 4, 5, 6 sequentially
for number in flat_stream:
print(number)
By using chain, you transform disparate data sources into a seamless pipeline, keeping your code memory-efficient and highly readable regardless of the volume of data.
Key Takeaway: Use itertools.chain to iterate over multiple collections as a single stream without the memory cost of creating temporary intermediate lists.