BiteSizedChunks.comLearn one small thing at a time.
Course contentsShow
← Linux
Lesson 786 of 1,8197. Storage Management and Logical VolumesFree lesson

Storage fault injection

Use system tools to inject I/O errors and verify service resilience.

In this lesson, you will learn how to intentionally simulate storage hardware failures to test how your applications and services handle I/O errors.

The Concept of Fault Injection

In a perfect world, hardware never fails. In reality, disks encounter bad sectors, intermittent connectivity issues, and controller timeouts. Storage fault injection allows you to verify if your services crash ungracefully or if they log errors and continue operating safely when the underlying block device stops responding.

Instead of physically unplugging cables, we use the Linux kernel’s fault-inject framework or the dm-error device mapper target. These tools allow us to "trick" the system into thinking a healthy disk is failing.

Simulating Errors with dm-error

The easiest way to perform fault injection is to wrap your storage device in a "mapped" device that returns errors for all read or write requests.

  1. Create the error device: First, identify your target device (e.g., /dev/sdb). You can create a mapping that immediately returns I/O errors:

    # Create a linear mapping to the device, but force errors
echo "0 $(blockdev --getsz /dev/sdb) error /dev/sdb 0" | dmsetup create faulty_disk

  2. Verify Resilience: Now, point your application or database to /dev/mapper/faulty_disk instead of the physical disk. When your application attempts to read or write, the kernel will return an EIO (Input/output error). Use this to observe your system logs:

    # Monitor system logs to see the application's response to the failure
journalctl -f

  3. Cleanup: Once you have confirmed that your application handles the error—or identifies where it fails to do so—you can remove the mapping:

    dmsetup remove faulty_disk

Why Do This?

By forcing these faults in a controlled test environment, you can verify your disaster recovery procedures without needing a real hardware catastrophe. It helps you identify if your mount points need specific errors=remount-ro flags, or if your application logic requires more robust retry loops for critical I/O operations.

Key Takeaway: Storage fault injection provides a safe, repeatable way to validate that your Linux services remain resilient when faced with inevitable hardware I/O failures.