Course contentsShow
← System Design
Lesson 448 of 1,91912. NoSQL: Wide-Column StoresFree lesson

Write Path: MemTable and Commit Logs

How writes go to an in-memory MemTable with WAL durability before being flushed to SSTables.

Write Path: MemTable and Commit Logs

What you'll learn: How BigTable (and similar wide-column stores) handle writes by using an in-memory buffer and a durable log to ensure fast performance and data safety.


The Write Path Problem

Wide-column stores need to handle massive write throughput while guaranteeing durability. Writing directly to disk on every operation would be too slow. The solution? Split the write path into two components that work together.


How It Works

When a write arrives in BigTable:

  1. Commit Log (Write-Ahead Log): The write is immediately appended to a sequential log file stored in GFS. This is fast because it's an append-only operation—no random disk seeks.

  2. MemTable (In-Memory Buffer): Simultaneously, the write is added to an in-memory data structure (the MemTable) sorted by row key and timestamp. This makes reads and subsequent writes blazingly fast.

The commit log provides durability—if a server crashes, the log can replay recent writes. The MemTable provides speed—all recent data lives in RAM for instant access.


Analogy

Think of a restaurant order system: the waiter writes your order on a paper ticket (commit log) that goes to the kitchen, while also entering it into a digital display (MemTable) for the chef. If the display crashes, the paper tickets ensure no orders are lost. If the chef needs to see current orders quickly, they check the display.


When MemTable Fills Up

Once the MemTable reaches a size threshold, it's flushed to disk as an immutable SSTable. The corresponding commit log entries can then be discarded since the data is now safely persisted in structured storage.


Key Takeaway: BigTable's write path uses commit logs for durability and MemTables for speed, ensuring both data safety and high-performance writes before eventually persisting to SSTables.