Parquet and Feather: Columnar Formats
You'll learn how to dramatically speed up data loading and saving with columnar file formats.
When working with large datasets, CSV and JSON files can become painfully slow. That's where columnar formats like Parquet and Feather shine. Unlike row-based formats (CSV), columnar formats store data by columns rather than rows, making them much faster for analytical workloads where you typically need subsets of columns across many rows.
What Are Parquet and Feather?
Parquet is a compressed, column-oriented format designed for efficiency. It's the industry standard for big data systems (Hadoop, Spark) and offers excellent compression ratios, reducing file sizes significantly.
Feather is a lightweight columnar format optimized for speed. It's less compressed than Parquet but extremely fast for reading and writing, making it ideal for temporary data storage or inter-process communication.
When to Use Each
- Parquet: Long-term storage, sharing data across tools, when file size matters
- Feather: Fast temporary storage, quick data exchange between Python processes
Basic Usage
import pandas as pd
df = pd.read_csv('large_data.csv')
# Write to Parquet (smaller file, slightly slower)
df.to_parquet('data.parquet')
df_parquet = pd.read_parquet('data.parquet')
# Write to Feather (larger file, very fast)
df.to_feather('data.feather')
df_feather = pd.read_feather('data.feather')
Both formats preserve data types perfectly (unlike CSV) and can be 10-100x faster for large datasets. Column selection is particularly efficient—you can read just the columns you need without loading the entire file into memory.
Key Takeaway: Use Parquet for storage and archival with compression, Feather for speed when working interactively—both dramatically outperform CSV for data science workflows.