Process Supervision

In this lesson, you will learn how to design a self-healing supervision tree that ensures your long-running Rust services remain operational through automated health checks and process restarts.

The Supervisor Pattern

In robust distributed systems, processes inevitably fail—whether due to memory leaks, unhandled panics, or external resource exhaustion. Instead of letting a service die permanently, we implement a Supervisor. The supervisor acts as a parent process (or a controller thread) that monitors the child’s exit status and health metrics. If the child fails, the supervisor restarts it according to a defined policy (e.g., exponential backoff).

Implementation Logic

To build a supervisor in Rust, you generally combine std::process::Command for lifecycle management with a communication channel to monitor health. Here is a simplified pattern:

use std::process::{Command, Child};
use std::time::{Duration, Instant};

fn supervise(mut child: Child) {
    loop {
        // Perform a health check
        if let Ok(Some(status)) = child.try_wait() {
            println!("Process exited with: {}. Restarting...", status);
            child = Command::new("./my_service").spawn().expect("Failed to restart");
        }
        
        // Sleep to prevent tight-looping on immediate failures
        std::thread::sleep(Duration::from_secs(1));
    }
}

Strategic Integration

While the code above handles the raw lifecycle, a production-grade supervisor must incorporate Health Checks. Relying solely on try_wait() is insufficient because a process might be "alive" (running) but deadlocked or unresponsive.

You should integrate a heartbeat mechanism:

1. Liveness Probe: The supervisor sends a ping over a Unix Domain Socket or a local pipe.
2. Graceful Termination: When a restart is needed, the supervisor should send a SIGTERM signal (using the nix crate or std::process) before force-killing the process.
3. Backoff: If the service fails repeatedly, the supervisor should increase the time between retries to avoid wasting CPU cycles on a service that is permanently broken.

By decoupling the "monitor" from the "worker," you create a resilient architecture where the supervisor remains simple and stable, while the worker can focus on complex business logic without fearing a permanent crash.

Key Takeaway: Process supervision shifts the burden of recovery from the application code to the infrastructure layer, ensuring high availability through reactive lifecycle management and proactive health monitoring.