Error Handling in Batch Pipelines
What you'll learn: How to design fault-tolerant batch pipelines that gracefully handle failures without losing work or corrupting data.
When processing thousands of documents or embedding requests in a batch pipeline, failures will happen—network timeouts, malformed documents, API errors, or resource exhaustion. The key is building pipelines that survive these failures without starting over or silently losing data.
Isolating Failures
Think of each item in your batch as independent cargo on a ship. If one crate is damaged, you don't sink the entire vessel. Wrap processing logic for each document or batch in try-except blocks:
def process_batch(documents):
results = []
for doc in documents:
try:
embedding = embed_document(doc)
results.append({"doc_id": doc.id, "embedding": embedding})
except Exception as e:
# Log but don't crash
log_error(f"Failed doc {doc.id}: {e}")
continue
return results
Dead-Letter Queues
Failed items shouldn't disappear. Save them to a "dead-letter queue"—a separate file or database table containing failures for manual review or retry:
def process_with_dlq(documents, dlq_path):
for doc in documents:
try:
process(doc)
except Exception as e:
write_to_dlq(dlq_path, {"doc": doc, "error": str(e)})
Partial Success Handling
Your pipeline should track which items succeeded. Even if 10 out of 1000 documents fail, you've still made 99% progress. Combine checkpointing (from lesson 485) with dead-letter queues to resume only failed items later.
Key Takeaway: Fault-tolerant batch pipelines isolate failures per item, preserve failed items in dead-letter queues for inspection, and treat partial success as progress—not failure.