Async Watchdog Patterns

In this lesson, you will learn how to build a robust watchdog monitor to detect and recover from stalled or "hung" asynchronous worker tasks.

The Watchdog Concept

In a complex system, a worker loop might stop making progress due to an infinite loop, a deadlock, or an unhandled await that never resolves. A watchdog pattern solves this by introducing a "heartbeat" mechanism: workers periodically report their health, and a dedicated monitor task tracks these reports. If a worker fails to signal within a predefined timeout, the watchdog assumes the task is dead and triggers a recovery routine, such as logging an error, restarting the task, or initiating a graceful shutdown.

Implementation Pattern

The most idiomatic way to implement this is using a tokio::sync::mpsc channel for heartbeat signals and a tokio::select! loop in the monitor.

async fn watchdog(mut receiver: mpsc::Receiver<()>, timeout: Duration) {
    loop {
        // The watchdog waits for a heartbeat or a timeout
        match tokio::time::timeout(timeout, receiver.recv()).await {
            Ok(Some(_)) => { /* Heartbeat received, worker is healthy */ }
            Ok(None) => break, // Channel closed
            Err(_) => {
                eprintln!("Worker timed out! Initiating recovery...");
                // Recovery logic goes here
            }
        }
    }
}

Key Considerations

1. Granularity: If your watchdog tracks multiple tasks, use a HashMap inside the monitor to map TaskId to a LastSeen timestamp. Check this map periodically rather than waiting for a single channel.
2. Backpressure: Use bounded channels for heartbeats. If a worker is so saturated that it cannot send a heartbeat, it likely needs to be restarted anyway; a full channel acts as a diagnostic signal that the worker is overwhelmed.
3. Recovery: Ensure your recovery logic doesn't introduce a "thundering herd" or unstable restart cycle. Always implement exponential backoff if the watchdog needs to restart a failed worker.

By delegating health monitoring to a separate, lightweight task, you decouple the supervision logic from the execution logic, ensuring your application remains resilient even when individual components fail.

Key Takeaway: The async watchdog pattern ensures system reliability by enforcing strict heartbeat deadlines, allowing you to proactively detect and recover from hung worker tasks before they impact the entire runtime.