In this lesson, you will learn how Apache Iceberg uses optimistic concurrency control (OCC) to maintain a consistent data state when multiple users read from and write to your table simultaneously.
The Challenge of Distributed Concurrency
In distributed data lakehouses, multiple processes often attempt to update or append to the same table at the same time. Without a mechanism to manage these conflicts, you risk "lost updates" or reading partially written data. Iceberg solves this using Optimistic Concurrency Control.
How Optimistic Concurrency Control Works
Instead of locking the entire table—which would stop all other reads and writes—Iceberg assumes that conflicts are rare. Every write operation follows a specific workflow:
- Snapshot Creation: The writer prepares the new data files and creates a new metadata file containing a pointer to these files.
- Commit Attempt: The writer attempts to update the table’s current metadata pointer to the new version.
- Conflict Check: Iceberg checks if the current metadata has changed since the writer began its process. If a different process successfully committed a change in the meantime, the current operation detects a conflict.
- Retry: If a conflict occurs, Iceberg rejects the commit, and the writer retries the operation by basing the new write on the updated current snapshot.
This allows your readers to keep querying the "last known good" version of the table without interruption, while writers ensure that every commit results in a consistent, atomic state.
-- Example of an ACID-compliant operation in Iceberg
-- Even if another process is writing to the table,
-- this UPDATE remains consistent.
UPDATE sales_data
SET status = 'processed'
WHERE event_date = '2023-10-27';
Because Iceberg verifies the table state at the moment of the commit, you never have to worry about a reader seeing a "half-written" transaction. The data is either fully committed and visible in a new snapshot, or it is not present at all. By delegating the conflict detection to the metadata layer, Iceberg ensures that your data integrity remains ironclad across distributed systems without sacrificing performance.
Key Takeaway: Optimistic Concurrency Control allows Iceberg to maintain data integrity by validating that no conflicting writes have occurred before committing a new snapshot, ensuring that concurrent readers always see a consistent, atomic version of the truth.