Performing Fault Injection Testing

In this lesson, you will learn how to systematically validate your redundancy and recovery mechanisms by deliberately introducing faults into your system.

The Philosophy of Fault Injection

Designing for safety is only half the battle; you must prove your mechanisms work when things go wrong. If you have implemented Triple Modular Redundancy or Software Checkpointing, you cannot wait for a cosmic ray to flip a bit to test if your system recovers correctly. Fault injection allows you to verify that your safety strategies—like Graceful Degradation or Watchdog Timers—actually trigger the intended recovery sequences.

Systematic Injection Techniques

To perform this effectively, you must target specific failure modes:

1. Software-Level Injection: You can manually alter variables or corrupt memory buffers during execution to simulate data corruption.
2. Hardware-Level Injection: This involves using tools like debuggers to modify registers, or physically manipulating pins (e.g., grounding a signal line) to simulate sensor or communication failures.

Consider a system using Triple Modular Redundancy (TMR) where three variables hold the same sensor state. To test if your voter logic works, you inject a fault into one:

// Simulated Fault Injection for TMR validation
void test_tmr_voting_mechanism() {
    sensor_data[0] = 100;
    sensor_data[1] = 100;
    sensor_data[2] = 100; // Correct states

    // Inject fault into the first voter input
    sensor_data[0] = 999; 

    // Verify recovery: Does the voter logic correctly identify 
    // the mismatch and output the majority (100)?
    int result = vote_on_sensors(sensor_data);
    assert(result == 100); 
}

Validating Recovery

When injecting faults, do not just check if the system survives. Validate the entire lifecycle:

• Detection: Did the system catch the error (e.g., via Error Detection Codes or Watchdog Timers)?
• Isolation: Did the system prevent the corrupted data from reaching your Fail-Safe Output States?
• Recovery: Did the system perform a Safe Recovery Sequence or revert to a Checkpoint?

If your system crashes or enters an undefined state, your redundancy isn't working as intended. By systematically applying these faults, you transform your assumptions about reliability into verifiable data.

Key Takeaway: Fault injection is the empirical proof that your safety mechanisms—such as TMR and checkpointing—are robust enough to maintain system integrity during actual hardware or software failures.