Handling drive failure and recovery

Execute safe disk removal and replacement procedures in production Linux systems.

In this lesson, you will learn how to safely remove a failing drive and prepare your system for the physical replacement process without risking data corruption.

The Logic of Safe Removal

When a drive begins showing signs of failure—such as I/O errors reported by your filesystem health monitoring—the priority is to stop the kernel from attempting to read from or write to that device. In a production Linux environment, you cannot simply unplug a drive, as the kernel may still hold open file handles or active mount points.

Before physical removal, you must perform a "graceful detach." This ensures that all buffered I/O is flushed to the disk and the block device stack is cleanly unmounted.

Step-by-Step Recovery Procedure

Flush and Unmount: First, ensure all applications have finished writing data. Use sync to push filesystem buffers to the disk, then unmount the filesystem:
```
sync
umount /dev/sdb1
```
Quiesce the Device: If you are using software RAID or a logical volume manager, you must logically remove the device from the array or volume group to prevent the kernel from attempting to re-access the "missing" device. If it is a standalone disk, ensure no process has an open file handle:
```
lsof /dev/sdb
```
Detach from the SCSI Subsystem: To prevent "ghost" devices or kernel panics when you physically pull the drive, inform the kernel that the device is going away. You can send a remove command to the SCSI host:
```
echo 1 > /sys/block/sdb/device/delete
```
Physical Replacement: Once the system confirms the device is deleted, it is safe to physically swap the drive.
Rescan: After the new drive is inserted, you don't need to reboot. You can tell the kernel to rescan the bus to pick up the new hardware:
```
echo "- - -" > /sys/class/scsi_host/host0/scan
```

By following this sequence, you treat the hardware as an extension of the kernel’s block device stack rather than a simple piece of peripheral equipment. This minimizes the risk of filesystem corruption during the transition.

Key Takeaway: Always ensure the kernel has released all filesystem locks and flushed buffered I/O before logically deleting a drive from the SCSI subsystem, as this prevents system instability during physical hardware replacement.