Supervision Trees for Resilient Systems

In this lesson, you will learn how to design hierarchical thread structures where parent threads act as supervisors to monitor and automatically restart failed children.

The Resilience Hierarchy

In complex systems, individual threads often crash due to unexpected inputs or resource exhaustion. Instead of letting the entire application collapse, we implement supervision trees. A supervisor thread spawns a child, keeps a reference to its handle, and monitors it. If the child thread terminates unexpectedly, the supervisor detects the exit status and triggers a restart logic.

Implementing a Basic Supervisor

Think of this as a "watchdog" pattern. The supervisor keeps the child alive by looping until a success condition is met or a max-retry limit is reached.

use std::thread;

fn supervisor() {
    loop {
        let child = thread::spawn(|| {
            // Simulate volatile, error-prone task
            if rand::random() { panic!("Child crashed!"); }
            println!("Task complete.");
        });

        match child.join() {
            Ok(_) => break, // Success
            Err(_) => {
                println!("Restarting child...");
                continue; // Trigger recovery
            }
        }
    }
}

Why Hierarchies Matter

When your system grows, you don't just have one supervisor. You build a tree:

1. The Root Supervisor: Manages top-level services (e.g., networking, database).
2. Intermediate Supervisors: Manage functional groups (e.g., a group of workers processing data).
3. Workers: The leaf nodes that perform the actual computation.

By separating the "monitoring logic" from the "business logic," you prevent state corruption. If a worker crashes, the supervisor can clean up associated resources—such as closing stale network sockets or resetting partial buffers—before re-spawning a fresh worker thread.

This structure ensures that failures are contained. If a leaf node fails, the supervisor handles the restart locally without alerting the root or other unrelated branches of the tree. This keeps your system predictably resilient, as the "blast radius" of any specific panic is restricted to its local branch.

Key Takeaway: Supervision trees provide fault isolation by delegating error recovery to parent threads, ensuring that local crashes lead to controlled restarts rather than system-wide termination.