Embedded Database Storage

In this lesson, you will learn how to bridge the gap between volatile application memory and persistent storage while upholding strict ACID (Atomicity, Consistency, Isolation, Durability) guarantees.

When building an embedded database in Rust, you aren't just writing to a file; you are constructing a state machine that must survive power loss. The core challenge is the "Write-Ahead Log" (WAL). Before modifying your main data structure (e.g., an LSM-Tree or B-Tree), you must serialize the operation to a log file on disk and force a flush to ensure durability.

Atomic Persistence via Fsync

If your process crashes between writing the WAL and updating the database file, you must be able to recover. In Rust, this requires careful use of std::fs::File::sync_all() to ensure data actually reaches the physical platter or flash controller. Without this, the OS-level page cache may discard your "writes" during a crash.

Consider this simplified atomic write pattern:

use std::fs::{File, OpenOptions};
use std::io::{Write, Result};

fn write_to_store(path: &str, data: &[u8]) -> Result<()> {
    // 1. Write to the WAL first
    let mut wal = OpenOptions::new().append(true).create(true).open("db.wal")?;
    wal.write_all(data)?;
    wal.sync_all()?; // Ensure durability before proceeding

    // 2. Perform the actual data update
    let mut db_file = OpenOptions::new().write(true).open(path)?;
    db_file.write_all(data)?;
    db_file.sync_all()?; 

    // 3. Clear/Checkpoint WAL
    Ok(())
}

Challenges of ACID Integration

• Atomicity: Even with sync_all, your logic must handle partial writes. If the system fails mid-write, the WAL might contain a half-written entry. You typically solve this by appending a checksum (like CRC32) to every log entry; during recovery, you discard any entries that fail verification.
• Consistency/Isolation: While the database file is being updated, readers might see inconsistent data. Advanced embedded engines often use Copy-on-Write (CoW) techniques, where new data is written to a new file/block, and a single "root pointer" is updated atomically at the very end of the process, effectively providing snapshots without heavy locking.

Key Takeaway: ACID guarantees in embedded storage rely on the strict ordering of I/O operations and the rigorous use of fsync to force buffers through the OS cache to persistent media.