Course contentsShow
← Python
Lesson 392 of 1,0606. Error Handling and File I/OFree lesson

Handling Large Datasets with Buffered I/O

Optimize performance when processing massive files by reading and writing data in smaller, manageable chunks rather than loading entire files into memory.

In this lesson, you will learn how to process massive files efficiently by reading and writing data in small, manageable chunks instead of loading everything into your computer's memory at once.

The Problem with "All-at-Once" Processing

When you open a small file, it is easy to use f.read() to bring the entire contents into a Python variable. However, if you are working with a gigabyte-sized log file, your computer might run out of RAM, leading to a crash or a sluggish system. Buffered I/O solves this by acting like a conveyor belt: you only handle a small "bucket" of data at a time.

Processing in Chunks

Instead of reading the whole file, you define a fixed buffer size (for example, 64 kilobytes) and read the file in a loop. This keeps your memory footprint low and constant, regardless of the total file size.

Here is how you can implement this pattern using a while loop:

# Define a chunk size (e.g., 64KB)
CHUNK_SIZE = 64 * 1024 

with open('massive_data.bin', 'rb') as source_file:
    with open('processed_data.bin', 'wb') as dest_file:
        while True:
            # Read a small chunk
            chunk = source_file.read(CHUNK_SIZE)
            
            # If chunk is empty, we've reached the end
            if not chunk:
                break
            
            # Process and write the chunk immediately
            dest_file.write(chunk)

Why This Works

The read(CHUNK_SIZE) method fetches only the amount of data you requested. By wrapping the process in a while loop, you ensure that as soon as one chunk is processed and written, it is discarded from memory to make room for the next one. This approach allows your Python script to handle files much larger than the physical memory available on your machine. Using this "read-process-write" cycle, you remain in control of your system resources while ensuring your program remains fast and stable.

Key Takeaway: To handle massive datasets, always process data in discrete, fixed-size chunks to maintain a low and predictable memory footprint.