Course contentsShow
← Cloud and Infrastructure
Lesson 2632 of 2,98956. Data Pipelines and StreamingFree lesson

File Formats for Analytics

Choosing between Parquet, ORC, Avro, and JSON for columnar storage and query performance in data lakes.

File Formats for Analytics

You'll learn: How to choose the right file format—Parquet, ORC, Avro, or JSON—for storing analytics data in your data lake based on query performance and storage efficiency needs.

Why File Format Matters in Data Lakes

When storing data in object storage (your data lake foundation), the file format dramatically impacts query speed, storage costs, and processing efficiency. Think of it like choosing between organizing books by topic (columnar) versus chronologically (row-based)—the right choice depends on how you'll search them.

The Four Main Contenders

Parquet is the most popular columnar format for analytics. It excels at queries that read specific columns from massive datasets (like "SELECT revenue FROM sales"), compresses exceptionally well, and works seamlessly across Spark, Athena, and BigQuery. Use it as your default for analytical workloads.

ORC (Optimized Row Columnar) is Parquet's cousin, optimized for Hive and other Hadoop ecosystems. It offers slightly better compression and faster reads in Hive-based environments but has narrower ecosystem support than Parquet.

Avro stores data row-by-row with embedded schemas, making it ideal for streaming pipelines and schema evolution. Choose Avro when writing data frequently or when your schema changes often—not for analytical queries.

JSON is human-readable and flexible but inefficient for analytics. It's uncompressed, row-based, and slow to parse. Use JSON only for small datasets, exploratory work, or when interoperability trumps performance.

Decision Framework

For analytical queries on large datasets: Parquet
For Hive-heavy environments: ORC
For streaming ingestion with evolving schemas: Avro
For exploration or APIs: JSON

Parquet typically reduces storage costs by 75-90% compared to JSON while accelerating queries by 10-100x through columnar scanning and predicate pushdown.

Key Takeaway: Use Parquet for analytics in data lakes—it balances query performance, compression, and ecosystem compatibility better than alternatives, making it the de facto standard for columnar storage.