In this lesson, you will learn how to manage multiple files simultaneously using nested context managers to ensure your program remains robust and memory-efficient.
The Challenge of Multiple Files
When working with multiple files—such as reading data from a source file and writing processed results to a destination file—you must ensure that every file handle is closed correctly. If an error occurs while writing to the second file, a poorly managed program might leave the first file "hanging" open. This leads to memory leaks and can exhaust the operating system's limit on open file descriptors, causing your program to crash.
Using Nested Context Managers
The cleanest way to handle multiple resources is to nest your with statements. Python’s with statement guarantees that the __exit__ method of the context manager is called as soon as the block terminates, regardless of whether the code finished successfully or raised an exception.
When you nest them, Python ensures an "inside-out" cleanup order: the inner file closes first, followed by the outer file.
from pathlib import Path
source = Path("data.txt")
destination = Path("output.txt")
try:
with source.open('r') as infile:
with destination.open('w') as outfile:
for line in infile:
# Perform your processing
processed_line = line.strip().upper()
outfile.write(processed_line + '\n')
except IOError as e:
print(f"An error occurred during file operations: {e}")
Why This Matters
By nesting these managers, you create a fail-safe hierarchy. Even if the outfile.write() operation triggers an error, the context manager mechanism catches the exit signal and closes outfile first, and then closes infile. This prevents resource exhaustion and keeps your file system state consistent. If you ever find yourself needing to manage more than three files at once, you can also use contextlib.ExitStack to manage these dynamic sets of file handles without creating deeply indented code, though simple nesting remains the most readable approach for standard operations.
Key Takeaway: Always nest your context managers when working with multiple files to guarantee that each handle is closed safely in the correct order, protecting your system from resource leaks.