Course contentsShow
← System Design
Lesson 763 of 1,91920. Data Pipelines and ETLFree lesson

Cost and Storage Efficiency

Storage costs, compute separation, and compression tradeoffs between lakes (cheap storage) and warehouses (faster queries).

Cost and Storage Efficiency

What you'll learn: How data lakes and data warehouses differ in storage costs, compute separation, and compression tradeoffs.

Core Idea

When designing data pipelines, one of your biggest decisions is balancing cost against query speed. Data lakes offer cheap storage at the expense of query performance, while data warehouses optimize for fast queries but cost more.

Storage Cost Fundamentals

Data lakes use object storage (like Amazon S3, Azure Blob Storage) which costs pennies per GB per month. Raw files sit compressed in formats like Parquet or ORC, and you only pay for what you store. Storage and compute are completely separated—you spin up query engines only when needed.

Data warehouses store data in optimized, indexed formats designed for instant queries. This comes with higher storage costs and often requires provisioned compute capacity that runs continuously or scales automatically (but still incurs costs).

The Compression Tradeoff

Both systems use compression, but differently:

  • Lakes: Highly compressed files (Parquet with Snappy/Gzip) minimize storage bills. The tradeoff? Queries must decompress and scan large files.
  • Warehouses: Compression is balanced with indexing and columnar organization for fast scans. Storage costs more, but queries run 10-100x faster.

Compute Separation

Data lakes excel at decoupling storage from compute. Your data sits in cheap object storage forever, and you query it with temporary clusters (Spark, Presto) that shut down when idle. Warehouses traditionally bundled storage with compute, though modern ones (Snowflake, BigQuery) now separate them too—but still at premium pricing.

Key Takeaway: Data lakes prioritize minimal storage costs with pay-per-query compute; warehouses trade higher storage and compute costs for consistently fast queries. Choose based on access patterns and budget constraints.