The Commit Log Abstraction
What you'll learn: How event streams model data as an append-only, ordered log that can be read from any position.
The Core Idea
At the heart of every event streaming platform lies a deceptively simple data structure: the commit log. This is an append-only sequence of records, where each new event gets added to the end with an ever-increasing offset (position number). Unlike a traditional message queue that deletes messages after consumption, a commit log keeps all events for a configurable retention period, allowing any consumer to read from any position at any time.
Think of it like a ledger in accounting: entries are written in order, never erased, and anyone can go back and audit previous entries. Each line has a sequential number (the offset), and you can replay the entire history from the beginning—or jump to any point you choose.
Why This Matters for Event Streaming
The commit log abstraction provides three critical capabilities:
- Durability: Events are written to disk and replicated, surviving crashes
- Ordering: Every event in a partition has a strict sequential position
- Replayability: Consumers can rewind to any offset and re-process events
This is fundamentally different from traditional message brokers. When you acknowledge a message in a queue, it's gone. But with a commit log, acknowledging just means "I've processed up to offset 147"—the data remains, and you can always go back to offset 100 if needed.
Multiple consumers can read the same log independently, each maintaining their own position. One consumer might be at offset 1000 while another is still catching up at offset 500.
Key Takeaway: The commit log abstraction transforms events into a durable, ordered, replayable sequence—providing the foundation for event streaming's power to serve as both a messaging system and a queryable record of what happened.