Async Workflow Persistence

In this lesson, you will learn how to design "Async Workflow Persistence" to ensure multi-step asynchronous processes survive system crashes and resume exactly where they left off.

When building complex async flows—like payment processing or multi-stage distributed data migrations—a system crash mid-execution can leave your data in an inconsistent "zombie" state. To prevent this, we move beyond simple in-memory futures and implement a Durable Task Queue.

The Persistence Pattern

Instead of running a long-lived async function that maintains state in memory, we break the workflow into discrete, idempotent steps. Each step is represented as a task in a persistent store (like a database or transactional log).

1. Transactionality: Before executing a step, mark it as "In Progress" in your database.
2. Idempotency: Ensure that if a step runs twice (due to a restart), the result remains the same.
3. Recovery: Upon startup, a "reconciliation loop" queries the database for tasks that are "In Progress" but have no active worker, automatically re-queueing them.

Minimal Implementation Example

Consider a process that needs to Reserve Inventory then Charge User. We store the state of the workflow in a database:

#[derive(serde::Serialize, serde::Deserialize)]
enum WorkflowStep { ReserveInventory, ChargeUser, Complete }

struct Task {
    id: Uuid,
    step: WorkflowStep,
}

async fn run_workflow(task: Task) {
    match task.step {
        WorkflowStep::ReserveInventory => {
            if reserve_inventory().await.is_ok() {
                // Persist the state transition before proceeding
                db::update_step(task.id, WorkflowStep::ChargeUser).await;
                run_workflow(Task { id: task.id, step: WorkflowStep::ChargeUser }).await;
            }
        }
        WorkflowStep::ChargeUser => {
            // If the system crashed here, the re-queue logic restarts at this step
            if charge_user().await.is_ok() {
                db::update_step(task.id, WorkflowStep::Complete).await;
            }
        }
        WorkflowStep::Complete => {}
    }
}

By persisting the WorkflowStep transition to a database before triggering the next await, you transform an ephemeral async execution chain into a fault-tolerant state machine. If your process crashes, the next instance of your application simply reads the "In Progress" record from the database and resumes from the specific WorkflowStep rather than the beginning.

Key Takeaway: Async workflow persistence is achieved by breaking long-running processes into idempotent, state-backed steps that commit progress to an external store before proceeding to the next asynchronous operation.