Managing Memory in Large Batch Jobs
You'll learn: How to prevent memory crashes when processing massive datasets by using generators, chunking, and garbage collection.
The Memory Problem
When embedding thousands of documents, naively loading everything into memory at once can crash your system. Imagine trying to carry all your groceries in one trip instead of making multiple trips—you'll drop everything. The same principle applies to data processing.
Three Essential Techniques
1. Generators (Stream, Don't Load)
Instead of loading all documents into a list, use Python generators that yield one item at a time. This processes data "on demand" without storing everything in RAM simultaneously.
# Bad: loads everything into memory
all_docs = [process(doc) for doc in million_documents]
# Good: processes one at a time
def doc_generator(documents):
for doc in documents:
yield process(doc)
2. Chunking (Process in Small Batches)
Break large datasets into manageable chunks. Process 100 documents, clear memory, then process the next 100. You've already used this for embedding batches—now apply it to the entire pipeline.
for i in range(0, len(documents), chunk_size):
chunk = documents[i:i + chunk_size]
embeddings = embed_batch(chunk)
save_to_database(embeddings)
3. Garbage Collection (Manual Cleanup)
Python's automatic garbage collector sometimes needs help. Explicitly delete large objects and trigger collection between batches.
import gc
process_chunk(chunk)
del chunk # Free memory
gc.collect() # Trigger cleanup
Why This Matters
These techniques let you process datasets larger than your available RAM—essential when working with enterprise-scale document collections or training data pipelines.
Key Takeaway: Use generators to stream data, chunk processing into small batches, and manually trigger garbage collection to handle datasets far exceeding your system's memory capacity.