Hardware Failure Models

In this lesson, you will learn how to categorize and architect recovery strategies for specific hardware failure modes, ranging from disk-level corruption to total node power loss.

When designing for high availability, we assume hardware will fail. We classify these failures by their blast radius and the recovery mechanisms required to maintain ACID properties.

1. Individual Disk Failures (Media Failure)

A disk failure results in the loss of a data block or an entire spindle. If the storage engine utilizes Write-Ahead Logging (WAL) and checksums, we can detect corruption at the page level.

• Strategy: Use Redundant Arrays of Independent Disks (RAID) or software-defined replication (like mirroring segments across disks). When a checksum mismatch occurs, the system fetches the valid page from a mirror or reconstructs it via parity, ensuring that Data Integrity is maintained without crashing the instance.

2. Controller Faults

If a disk controller fails, the entire I/O subsystem attached to that controller becomes unreachable, potentially leading to a "hanging" state or kernel panic.

• Strategy: This necessitates path redundancy. By using multipath I/O, the database can reroute traffic to an alternative controller. If the controller failure causes a write timeout, the system must trigger a synchronous commit failure to prevent partial data writes that would violate durability guarantees.

3. Node Power Loss (Crash Recovery)

Power loss is the most common "volatile" failure. Because RAM is cleared, the system faces an inconsistency between the data residing on stable storage and the in-flight transactions that were never flushed from the buffer pool.

• Strategy: We rely on the Checkpointing mechanics and the WAL. Upon restart, the database engine initiates a recovery process:

  -- Recovery flow logic
  REDO (log_sequence_number); -- Replay committed ops not yet in data files
  UNDO (transaction_id);       -- Roll back uncommitted ops during crash
  

By replaying the WAL, the database moves to the last known consistent state defined by the Checkpoint, effectively ignoring the volatile state lost during the power event.

Key Takeaway: Recovery design is a layered hierarchy where local checksums handle disk rot, multipath controllers handle transit failure, and WAL-based replay provides the final barrier against volatile memory loss.