In this lesson, you will learn how to traverse nested directory structures to identify and process files using Python’s path manipulation tools.
Recursive Traversal with pathlib
When you need to perform an action on every file within a directory and all of its subdirectories—such as searching for all .log files in a deep project structure—manual recursion becomes cumbersome. Instead, we use the Path.rglob() method. This method implements a "recursive glob," which walks the directory tree starting from your base path and yields matching files as it discovers them.
Because rglob returns a generator, it is memory-efficient; it does not load the entire list of files into RAM at once, which is perfect for processing large filesystems.
Implementation Example
Suppose you want to find and print the path of every .txt file inside a folder named data:
from pathlib import Path
def process_text_files(base_directory):
base = Path(base_directory)
# rglob handles the recursion automatically
for file_path in base.rglob("*.txt"):
print(f"Found file: {file_path.name} in {file_path.parent}")
# Usage
process_text_files("data")
The string passed to rglob() acts as a filter. Using "*" would capture every file, while "*.txt" filters specifically for text documents. If you need to perform more complex logic—such as checking file metadata (like size or creation date) retrieved in previous lessons—you can simply add conditional logic inside the for loop.
Iterative Scanning vs. Manual Recursion
While you could write a function that calls itself to enter each subdirectory (often using Path.iterdir() and checking is_dir()), rglob is the idiomatic, standard library way to handle this. It abstracts away the stack-management of recursion and the error handling required for permission-restricted folders. By using these declarative patterns, your code remains clean, readable, and highly performant across different operating systems.
Key Takeaway: Use Path.rglob() for efficient, declarative recursive searching that abstracts away the complexity of walking deep directory trees.