In this lesson, you will learn how to drastically reduce the amount of data read from disk by using zone maps to skip irrelevant file blocks.
The Problem: Scanning Everything
When you store data in a columnar format, you already benefit from reading only the columns you need. However, without additional metadata, your system still has to scan every single block within those columns to check if the data matches your query criteria. If you are looking for user_id = 500 in a file containing millions of rows, reading every byte is an expensive I/O operation.
The Solution: Zone Maps
A Zone Map is a metadata structure—usually stored in the footer or header of a columnar file—that tracks the minimum and maximum values for specific columns within each data block.
Think of it like a table of contents for a book. Instead of reading every page to find a specific chapter, you check the index first. When a query arrives with a predicate (a filter condition), the engine performs predicate pushdown:
- The engine checks the Zone Map for a block.
- If your query asks for
value > 1000and the Zone Map says the block’smaxis500, the engine knows it is mathematically impossible for that block to contain your data. - The engine skips that entire block, saving precious I/O cycles.
Conceptual Illustration
Imagine a file split into three blocks with these ranges:
- Block 1:
[min: 1, max: 100] - Block 2:
[min: 101, max: 200] - Block 3:
[min: 201, max: 300]
If you run SELECT * WHERE val > 250, the query engine inspects the map:
- Block 1:
max 100<250? Yes. Skip. - Block 2:
max 200<250? Yes. Skip. - Block 3:
max 300>=250? Yes. Read block.
By using these min/max values, we avoid the overhead of opening and decompressing blocks that contain no relevant records, keeping our read performance high and our resource consumption low.
Key Takeaway: Zone Maps enable data skipping by maintaining min/max metadata for blocks, allowing query engines to ignore entire chunks of data that do not satisfy the filter predicate.