Partial Failure Recovery
What you'll learn: How to salvage successful work when only part of your data pipeline fails, avoiding wasteful reruns from scratch.
The Problem
In complex data pipelines, you might process 100 files or run 20 transformation steps. What happens when step 15 fails but steps 1–14 succeeded perfectly? Without recovery mechanisms, you'd restart everything—wasting compute, time, and money.
Core Concept
Partial failure recovery uses checkpoints and transaction boundaries to save progress at strategic points. When failure strikes, you resume from the last successful checkpoint rather than the beginning.
Think of it like a video game with save points. If you die on level 5, you don't restart from level 1—you reload from the last checkpoint.
How It Works
Checkpoints mark completed work:
- Store intermediate results (processed file IDs, completed row ranges)
- Record state in a persistent location (database, file system)
- Enable the pipeline to query "What have I already done?"
Transaction boundaries define atomic units of work:
- Group related operations (e.g., "process one file completely")
- Either all operations in the boundary succeed, or none persist
- Prevent half-processed, corrupted data
Implementation Pattern
When your pipeline runs:
- Check the checkpoint store: "What's already complete?"
- Skip completed work
- Process remaining items
- After each successful unit, write a checkpoint
- If failure occurs, next run resumes from last checkpoint
This pairs perfectly with idempotency (you've learned) and retry strategies—retries handle transient failures, checkpoints handle permanent ones or partial batch failures.
Key Takeaway: Partial failure recovery uses checkpoints to save progress and transaction boundaries to ensure data consistency, letting pipelines resume mid-stream rather than starting over from scratch.