RAID array failover testing

Simulate component failures to validate redundancy and alerting mechanisms.

In this lesson, you will learn how to proactively validate your data redundancy by manually simulating disk failures in a software RAID array.

Why Test Failover?

Even if you have configured RAID-1 or RAID-6 correctly, you shouldn't wait for a real hardware death to find out if your system responds as expected. "Failover testing" is the process of intentionally marking a drive as faulty to ensure that the array continues to serve data, the kernel generates the appropriate alerts, and your system remains operational.

Simulating a Failure

Using mdadm, you can simulate a drive failure without physically unplugging a cable. First, identify your RAID device (e.g., /dev/md0) and the active member drives. You can mark a drive as "failed" using the --fail flag:

# Mark /dev/sdb1 as faulty in array /dev/md0
sudo mdadm --manage /dev/md0 --fail /dev/sdb1

Once you run this command, the kernel will immediately stop using that drive. The RAID array will enter "degraded" mode. You can check the status of your array with cat /proc/mdstat. You should see the array still active, but the failed device marked with a (F).

Observing the System Response

This is the most critical part of your test. While the array is in a degraded state:

Verify Availability: Ensure your applications can still read and write files on the mount point.
Check Alerts: Verify that your system’s monitoring tools (like mdadm --monitor or integrated log watchers) have triggered an alert. If you don't receive a notification, your alerting configuration needs adjustment.
Recovery: To return the system to a healthy state, remove the failed drive and re-add it (or replace it):

# Remove the faulty drive
sudo mdadm --manage /dev/md0 --remove /dev/sdb1

# Re-add the drive (or a fresh replacement)
sudo mdadm --manage /dev/md0 --add /dev/sdb1

By performing these steps, you confirm that your "safety net" is functional and that your automated recovery procedures—such as rebuild priority—work exactly as you defined them during your configuration phase.

Key Takeaway: Never assume your RAID redundancy works until you have manually triggered a simulated failure and confirmed that both your data accessibility and your alerting systems function correctly.