Course contentsShow
← Python
Lesson 714 of 1,06011. Standard Library EssentialsFree lesson

Walking Directory Trees for Recursive Processing

Implement recursive directory traversal patterns to locate specific files within nested folder structures using globbing and iterative path scanning techniques.

In this lesson, you will learn how to traverse nested directory structures to identify and process files using Python’s path manipulation tools.

Recursive Traversal with pathlib

When you need to perform an action on every file within a directory and all of its subdirectories—such as searching for all .log files in a deep project structure—manual recursion becomes cumbersome. Instead, we use the Path.rglob() method. This method implements a "recursive glob," which walks the directory tree starting from your base path and yields matching files as it discovers them.

Because rglob returns a generator, it is memory-efficient; it does not load the entire list of files into RAM at once, which is perfect for processing large filesystems.

Implementation Example

Suppose you want to find and print the path of every .txt file inside a folder named data:

from pathlib import Path

def process_text_files(base_directory):
    base = Path(base_directory)
    
    # rglob handles the recursion automatically
    for file_path in base.rglob("*.txt"):
        print(f"Found file: {file_path.name} in {file_path.parent}")

# Usage
process_text_files("data")

The string passed to rglob() acts as a filter. Using "*" would capture every file, while "*.txt" filters specifically for text documents. If you need to perform more complex logic—such as checking file metadata (like size or creation date) retrieved in previous lessons—you can simply add conditional logic inside the for loop.

Iterative Scanning vs. Manual Recursion

While you could write a function that calls itself to enter each subdirectory (often using Path.iterdir() and checking is_dir()), rglob is the idiomatic, standard library way to handle this. It abstracts away the stack-management of recursion and the error handling required for permission-restricted folders. By using these declarative patterns, your code remains clean, readable, and highly performant across different operating systems.

Key Takeaway: Use Path.rglob() for efficient, declarative recursive searching that abstracts away the complexity of walking deep directory trees.