BiteSizedChunks.comLearn one small thing at a time.
Course contentsShow
← Data Engineering
Lesson 213 of 8253. Storage Formats and File SystemsFree lesson

Understanding ORC Stripe and Footer Layouts

Deconstruct the ORC file structure to learn how stripes and footers facilitate rapid data access and schema evolution in large-scale datasets.

In this lesson, you will learn how the ORC file structure uses a hierarchical design of stripes and footers to enable high-performance data retrieval and flexible schema management.

While row-oriented formats require scanning an entire file to find specific values, the Optimized Row Columnar (ORC) format organizes data into logical units called stripes. A stripe is essentially a large, self-contained batch of rows (typically 250MB) that acts as the primary unit of read access. By breaking a file into these chunks, ORC allows distributed systems to process parts of a dataset in parallel without needing to read the entire file.

Each stripe contains three critical components:

  1. Index Data: Stores min/max statistics for each column, allowing the query engine to skip entire stripes if the data doesn't match the filter criteria (e.g., if you are looking for ID > 1000 and the stripe max is 500).
  2. Row Data: The actual columnar data, where each column is stored in its own stream, enabling efficient reading of only the required columns.
  3. Stripe Footer: A local directory containing the offsets for the index and row data streams within that stripe.

At the end of the entire ORC file sits the File Footer. Think of this as the "table of contents" for the whole file. It contains the schema information, the number of rows, and—most importantly—the list of stripes along with their byte offsets.

[ File Header ]
[ Stripe 1 (Index + Data + Footer) ]
[ Stripe 2 (Index + Data + Footer) ]
[ ... ]
[ File Footer (Metadata + Schema + Stripe Directory) ]
[ Postscript (Compression Info + Footer Length) ]

When a query engine opens an ORC file, it jumps immediately to the end to read the Postscript and File Footer. Because the footer tells the engine exactly where each stripe begins and which columns are stored where, the system can perform "projection" (reading only specific columns) and "predicate pushdown" (skipping unnecessary stripes) before a single byte of actual data is loaded into memory. This layout is what allows ORC to scale across petabytes while keeping I/O costs minimal.

Key Takeaway: ORC uses stripes for parallel, granular data access and a file-level footer to provide a map of the data, enabling efficient skipping of irrelevant columns and rows during query execution.