In this lesson, you will learn how the hierarchical structure of Iceberg metadata tracks the state of your data through snapshots, manifests, and data files.
The Anatomy of an Iceberg Table
Unlike traditional Hive tables that rely on directories to define state, Apache Iceberg acts as a database-style table format that manages state via a sophisticated metadata hierarchy. You can think of this structure as a "tree" that starts at the top and branches down to the raw data files.
The Metadata Hierarchy
The hierarchy is composed of three distinct layers:
- The Metadata File: This is the "brain" of the table. It stores the current table schema, partitioning configuration, and a list of snapshots. Each snapshot represents the state of the table at a specific point in time.
- Manifest Lists: Each snapshot points to a "Manifest List" file. This file tracks the collection of manifest files associated with that specific snapshot. It stores high-level statistics about those manifests, such as partition ranges, allowing query engines to prune entire chunks of data without opening the underlying files.
- Manifest Files: These are the workers of the metadata layer. A manifest file contains the actual list of data files (Parquet, Avro, or ORC). Crucially, it tracks metrics for each file—like min/max values for columns—which enables data skipping.
Conceptual Visualization
When you run a query, the engine reads the metadata hierarchy to identify exactly which files are needed:
[Metadata File]
│
[Snapshot]
│
[Manifest List]
│
[Manifest File A] ----> [Data File 1, Data File 2]
[Manifest File B] ----> [Data File 3]
By traversing this path, the query engine ignores data that doesn't match your WHERE clauses before a single row of actual data is read. Because this structure is immutable, Iceberg provides snapshot isolation, meaning readers are never blocked by writers; they simply point to an older version of the metadata tree while the writer creates a new branch.
Key Takeaway: Iceberg separates physical data storage from its logical state using a three-tier metadata hierarchy (Metadata File → Manifest List → Manifest File) that enables efficient query planning, data skipping, and time travel.