Defining the Six Dimensions of Data Quality

In this lesson, you will learn how to quantify the reliability of your data pipelines by applying the six foundational dimensions of data quality.

As data engineers, building robust pipelines—even those with exactly-once semantics or optimized checkpointing—is useless if the data itself is untrustworthy. We evaluate data quality through these six pillars:

1. Accuracy: Does the data reflect the real-world entity? For instance, does a sensor_reading accurately represent the actual physical temperature?
2. Completeness: Is the required data present? A dataset missing 30% of its user_id fields fails the completeness test, as it creates gaps in downstream aggregations.
3. Consistency: Does the data agree across different systems? If your streaming sink reflects a balance of $100 but your analytical warehouse reflects$110 for the same account, your system lacks consistency.
4. Timeliness: Is the data available when needed? In real-time streaming, if your event_time is delayed beyond your defined SLAs, the data loses its utility.
5. Validity: Does the data follow defined formats and business rules? For example, a transaction_amount must be a positive float, not a negative string.
6. Uniqueness: Are there redundant records? Duplicate events caused by retries in your distributed sinks can skew metrics like unique_page_views.

Practical Evaluation

To establish a baseline, you should implement validation logic within your processing layer or as a gatekeeper before your analytical store:

-- Example of a basic quality check in a SQL-based transformation
SELECT 
    COUNT(*) AS total_records,
    SUM(CASE WHEN user_id IS NULL THEN 1 ELSE 0 END) AS missing_users,
    SUM(CASE WHEN transaction_amount < 0 THEN 1 ELSE 0 END) AS invalid_amounts
FROM streaming_sink_table;

By monitoring these metrics, you shift from reactive debugging to proactive data governance. When a pipeline produces invalid data, these dimensions help you pinpoint whether the issue lies in upstream ingestion, transformation logic, or sink connectivity.

Key Takeaway: Data quality is not a binary state but a measurable spectrum; by monitoring accuracy, completeness, consistency, timeliness, validity, and uniqueness, you transform raw streams into reliable business assets.