Course contentsShow
← Python
Lesson 362 of 1,0606. Error Handling and File I/OFree lesson

Navigating Directory Hierarchies and Listing Contents

Learners will iterate through directory structures using generators to efficiently filter, search, and list files based on specific patterns.

In this lesson, you will learn how to use generators to efficiently navigate complex directory structures and filter files based on custom patterns.

When dealing with large file systems, loading every single file name into a list at once can be memory-intensive. Instead, we can use pathlib combined with generator expressions to "yield" files one by one as we find them. This allows your program to remain responsive even when scanning thousands of files.

Efficient Filtering with Generators

To explore a directory hierarchy, we use the rglob() method from pathlib.Path. When combined with a generator expression, we can define complex filtering logic that executes lazily.

Imagine you want to find all .log files in a project that are larger than a certain size. Instead of creating a massive list in memory, you can stream them:

from pathlib import Path

def get_large_log_files(directory, min_size_bytes):
    path = Path(directory)
    # This is a generator expression
    return (
        file for file in path.rglob("*.log") 
        if file.is_file() and file.stat().st_size > min_size_bytes
    )

# Usage
try:
    large_logs = get_large_log_files("./logs", 1024 * 1024)
    for log_path in large_logs:
        print(f"Found large file: {log_path.name}")
except PermissionError:
    print("Access denied to a subdirectory.")

Why Use This Approach?

By returning a generator object, the code inside the parentheses is not executed until you actually iterate over it with the for loop. If you only need to process the first few items or break early, the program never wastes time scanning the rest of the directory tree.

Notice how this interacts with our error handling knowledge: because the generator is lazy, errors like PermissionError (which might occur when accessing a restricted subdirectory) will only be raised at the exact moment the iteration reaches that specific path. This allows you to place your try-except blocks exactly where the traversal occurs, ensuring your script handles protected folders gracefully without crashing the entire search process.

Key Takeaway: Using generator expressions with rglob() allows you to traverse deep directory structures memory-efficiently, enabling you to process files on-demand while applying custom filters.