Course contentsShow
← System Design
Lesson 764 of 1,91920. Data Pipelines and ETLFree lesson

Data Governance and Quality

Enforcing data quality, lineage, and compliance in warehouses vs managing raw data sprawl in lakes.

Data Governance and Quality

You'll learn how data warehouses enforce quality and lineage versus how data lakes handle raw data sprawl—and why governance matters for both.

Core Idea

Data warehouses and data lakes take fundamentally different approaches to data quality and governance. Warehouses enforce structure and quality before data enters (schema-on-write), while lakes accept anything first and worry about quality later (schema-on-read).

Warehouses: Strict Control

In a warehouse, data governance is built into the pipeline:

  • Schema enforcement: Only data matching the defined schema gets in
  • Quality checks: ETL processes validate, clean, and transform data upfront
  • Lineage tracking: Clear documentation of where data came from and how it was transformed
  • Access controls: Role-based permissions on tables and columns
  • Compliance: GDPR, HIPAA rules enforced at ingestion

Think of it like a library catalog system—every book is classified, indexed, and shelved properly before anyone can access it.

Lakes: Flexible Sprawl

Data lakes accept raw data first, ask questions later:

  • No upfront validation: Any format, any source, dumped into storage
  • Data sprawl: Multiple copies, outdated versions, unknown lineage
  • Quality issues: Garbage in, garbage out—discovered only when queried
  • Governance challenges: Hard to track what's sensitive, who owns it, or if it's compliant
  • Post-hoc governance: Tools like data catalogs and metadata layers added later to organize the chaos

The lake is like a warehouse full of unlabeled boxes—everything's there, but finding trustworthy data requires detective work.

The Tradeoff

Warehouses trade flexibility for trust. Lakes trade governance for speed and cost. Modern solutions try to split the difference with "data lakehouse" patterns that layer warehouse-style governance onto lake storage.

Key Takeaway:
Warehouses enforce data quality and lineage upfront through schema and ETL validation; lakes accept raw data freely but face governance challenges from unstructured sprawl.