Testing Orchestrated Workflows

Testing Orchestrated Workflows: Simulating Failure

In this lesson, you will learn how to build integration tests that simulate failure at various stages of a long-running distributed process to ensure your orchestrator handles interruptions gracefully.

When dealing with orchestrated workflows—especially those involving Sagas or state machines—a "happy path" test is rarely enough. A true test suite for distributed systems must verify that your state persists correctly even when a worker crashes mid-task. If a workflow fails after an external API call but before updating the database, your system must know how to recover, retry, or compensate.

The Strategy: Controlled Interruption

To test these scenarios, you should implement "fault injection" points within your test environment. Instead of relying on random network errors, you can use a mock worker that triggers a controlled failure based on a specific input or state.

Consider a workflow that executes ReserveInventory followed by ProcessPayment. You want to test if the orchestrator resumes correctly if the process dies exactly between these two steps:

// Test snippet: simulating a mid-workflow crash
it('should recover when a worker crashes after inventory reservation', async () => {
  const workflowId = 'order-123';
  
  // 1. Force a failure specifically after step 1
  mockWorker.setFailPoint('ReserveInventory', { 
    after: 'success', 
    crashProcess: true 
  });

  // 2. Trigger the workflow
  await orchestrator.start(workflowId);

  // 3. Verify state persisted in the DB
  const state = await db.getWorkflowState(workflowId);
  expect(state.status).toBe('WAITING_FOR_PAYMENT');

  // 4. Simulate a new worker picking up the crashed task
  await worker.resume(workflowId);
  
  const finalState = await db.getWorkflowState(workflowId);
  expect(finalState.status).toBe('COMPLETED');
});

Why This Matters

By forcing these failures, you validate your State Persistency logic. If the workflow state isn't accurately captured in your database before the task is dispatched, a crash results in a "zombie" task. These tests force you to ensure that every worker is idempotent and that the orchestrator can safely re-drive a workflow from its last saved checkpoint without duplicating side effects.

Key Takeaway: Integration tests for orchestrated workflows must verify recovery by explicitly injecting failures between state transitions to ensure the system consistently resumes from the last known good checkpoint.