Course contentsShow
← Machine Learning and Deep Learning
Lesson 3054 of 3,53866. Production Monitoring and Data DriftFree lesson

Duplicate Detection and Data Integrity

Identify duplicate records, referential integrity issues, and data inconsistencies across sources.

Duplicate Detection and Data Integrity

What you'll learn: How to identify duplicate records, referential integrity issues, and data inconsistencies across sources in production ML systems.

Why Data Integrity Matters in Production

When your ML model serves predictions in production, data quality goes beyond statistical properties. Data integrity ensures that records are unique, relationships between entities are valid, and information remains consistent across different data sources. Unlike schema violations or range checks, integrity issues are about logical correctness.

Core Integrity Problems

Duplicate Detection

Duplicate records corrupt training data and bias predictions. In production, duplicates can arise from:

  • Multiple upstream sources sending the same event
  • Retry logic creating duplicate entries
  • Join operations producing Cartesian products

Detection approaches:

  • Exact duplicates: Hash-based deduplication using all or key fields
  • Near-duplicates: Similarity measures (edit distance, fuzzy matching) for records that should be unique but have slight variations
  • Temporal duplicates: Same entity appearing multiple times within a time window

Referential Integrity

Like database foreign keys, ML data often has relationships. A transaction should reference a valid customer ID; a product review should link to an existing product. Broken references indicate data pipeline failures.

Common checks:

  • Foreign key validation against reference tables
  • Orphaned records detection
  • Consistency of IDs across joined datasets

Cross-Source Consistency

When features come from multiple systems (databases, APIs, data lakes), the same entity might have conflicting values. Monitor for discrepancies in overlapping fields to catch synchronization issues or stale data.

Key Takeaway: Data integrity monitoring detects logical errors—duplicates, broken relationships, and cross-source inconsistencies—that schema and statistical checks miss but can severely degrade model performance and business outcomes.