In this lesson, you will learn to categorize data health metrics into five foundational pillars to establish a robust baseline for monitoring your data systems.
While data quality confirms if data is correct, data observability ensures we know why it might be broken. By organizing your monitoring strategy into these five pillars, you shift from reactive firefighting to proactive system health management:
1. Freshness
Freshness measures the "recency" of your data. It tracks whether data is arriving on time or if your pipeline has stalled. You monitor this by comparing the event_time of the most recent record against the current system time.
2. Volume
Volume tracks the quantity of records processed. An unexpected spike or sudden drop in record counts often indicates an upstream source issue or a silent failure in a distributed job.
3. Distribution
Distribution metrics look at the mathematical health of your fields. If an amount column suddenly shows a mean value 10x higher than its historical baseline, or if a categorical field displays an unexpected null-value spike, your data's statistical properties have drifted.
4. Schema
Schema metrics monitor the structural integrity of your data. This involves alerting on unexpected changes, such as a new column being added, a data type changing (e.g., int to string), or a field being dropped entirely, which could break downstream analytical models.
5. Lineage
Lineage connects the dots between datasets. It maps the dependencies of your data, allowing you to trace the "blast radius" of a failure. If a source table is corrupted, lineage tells you exactly which downstream dashboards and reports are compromised.
Implementation Example You can define these in a monitoring config to establish your baseline:
metrics:
freshness: { threshold_minutes: 15 }
volume: { expected_range: [1000, 5000] }
distribution: { column: "price", max_null_percentage: 0.01 }
schema: { enforce_strict: true }
lineage: { track_upstream: ["raw_logs", "enriched_events"] }
By standardizing your observability across these five dimensions, you create a holistic view of your data ecosystem. This framework ensures that whether a problem is structural or temporal, your monitoring stack provides the necessary context to identify the root cause immediately.
Key Takeaway: Categorizing metrics into freshness, volume, distribution, schema, and lineage provides a comprehensive observability framework that moves beyond simple quality checks to provide actionable insights into your entire data pipeline’s health.