Simulating Fault Injection

In this lesson, you will learn how to systematically test your system’s resilience by programmatically injecting failures into your network and hardware abstractions.

When building robust systems, standard integration tests are often insufficient because they assume a "happy path" where network sockets always connect and hardware operations always succeed. To test true resilience, you must introduce non-determinism during development to see how your application recovers from partial failures.

The Strategy: Interface-Driven Injection

The most effective way to simulate faults in Rust is through trait-based dependency injection. By abstracting your system calls behind a trait, you can swap the real implementation for a "Faulty" wrapper that randomly returns Err results or introduces artificial latency.

Consider a simple hardware sensor interface:

trait Sensor {
    fn read_temp(&self) -> Result<f32, String>;
}

struct FaultySensor<T: Sensor> {
    inner: T,
    failure_rate: f64, // 0.0 to 1.0
}

impl<T: Sensor> Sensor for FaultySensor<T> {
    fn read_temp(&self) -> Result<f32, String> {
        if rand::random::<f64>() < self.failure_rate {
            return Err("Hardware Timeout".to_string());
        }
        self.inner.read_temp()
    }
}

Implementing Failures

When you integrate this into your system, you shouldn't just inject random errors; you should inject meaningful failures that reflect real-world scenarios:

1. Network Partitioning: In your network wrapper, simulate a "hanging" request by sleeping for an excessive duration or closing the stream abruptly.
2. Resource Exhaustion: Mock your memory or disk allocators to return an error when a specific threshold is reached to verify that your cleanup logic triggers correctly.
3. Data Corruption: Inject bit-flips in your FFI boundaries to ensure your error-handling code catches invalid headers or corrupted structs before they lead to memory safety violations.

By controlling the environment, you transform "flaky" production bugs into reproducible test cases. Instead of hoping your code handles a dropped connection, you force the connection to drop, ensuring your Result pattern unwinds safely or your retry logic correctly triggers. This makes resilience an explicit property of your system, verified at every build.

Key Takeaway: Use trait-based abstractions to wrap production components with fault-injection logic, allowing you to force failures during testing to verify that your error-recovery paths are robust.