HBase Write Path and WAL
What You'll Learn
You'll understand how HBase ensures data durability during writes using a Write-Ahead Log before storing data in memory.
The Write Path Journey
When a write request arrives at HBase, data doesn't go directly to disk—it takes a carefully orchestrated path designed for both speed and safety.
Think of it like a restaurant kitchen: When a customer orders food, the waiter first writes the order on a ticket (backup) before shouting it to the chef (active processing). If the chef forgets, the written ticket ensures nothing is lost.
The Three-Step Write Process
-
Write-Ahead Log (WAL): First, the write is appended to the WAL, a sequential log file stored on disk. This ensures durability—even if the server crashes immediately after, the write is preserved. The WAL acts as HBase's safety net.
-
MemStore: After the WAL confirms the write, data is written to the MemStore, an in-memory buffer for each column family. This provides fast read access to recent writes without hitting disk.
-
Flush to HFiles: When the MemStore fills up (typically 128MB-256MB), it's flushed to disk as an immutable HFile. This converts the temporary in-memory data into permanent storage.
Why This Matters
The WAL is critical for crash recovery. If a Region Server fails, HBase can replay the WAL to reconstruct any MemStore data that hadn't been flushed yet. Without it, all in-memory writes would be lost.
The MemStore enables write batching—many small writes accumulate in memory and flush together efficiently, avoiding the overhead of many tiny disk writes.
Key Takeaway: HBase writes data to the Write-Ahead Log first for durability, then to the in-memory MemStore for speed, and finally flushes to permanent HFiles on disk—balancing safety, performance, and storage efficiency.