In this lesson, you will learn how to implement "Saga" patterns to roll back completed steps when a complex background workflow fails midway through execution.
When a background job consists of multiple steps—like reserving inventory, charging a credit card, and shipping a package—a failure in the final step leaves your system in an inconsistent state. If the shipment fails but the payment was already captured, you have a partial failure. To fix this, we use compensation logic.
The Compensation Pattern
The core idea is to treat every successful step as a reversible action. For every "do" action (e.g., chargeCard), you must define an "undo" action (e.g., refundCard). When a step in your worker fails, you catch the error and execute the inverse of every previously completed step in reverse order.
Consider this worker logic:
async function processOrder(order) {
const history = []; // Track completed steps
try {
await inventoryService.reserve(order.items);
history.push('RESERVE');
await paymentService.charge(order.total);
history.push('CHARGE');
await shippingService.createLabel(order);
} catch (error) {
console.error("Workflow failed, rolling back...");
// Reverse the steps
for (const step of history.reverse()) {
if (step === 'CHARGE') await paymentService.refund(order.total);
if (step === 'RESERVE') await inventoryService.release(order.items);
}
throw error; // Propagate for retries or Dead Letter Queues
}
}
Why it Matters
This approach ensures eventual consistency. Instead of leaving orphaned records in your database or money captured for unshipped goods, your worker acts as a self-correcting system. Note that your "undo" logic should ideally be idempotent; if your rollback itself fails, you may need to log it to a Dead Letter Queue for manual intervention or a secondary cleanup job.
By maintaining a history stack, you ensure that even if the workflow is deep and complex, you only revert the operations that actually succeeded, preventing unnecessary side effects during the recovery phase.
Key Takeaway: To handle partial failures, maintain a log of successful operations and execute their corresponding inverse actions in reverse order whenever a subsequent step in the workflow fails.