Duplicate Detection and Data Integrity
What you'll learn: How to identify duplicate records, referential integrity issues, and data inconsistencies across sources in production ML systems.
Why Data Integrity Matters in Production
When your ML model serves predictions in production, data quality goes beyond statistical properties. Data integrity ensures that records are unique, relationships between entities are valid, and information remains consistent across different data sources. Unlike schema violations or range checks, integrity issues are about logical correctness.
Core Integrity Problems
Duplicate Detection
Duplicate records corrupt training data and bias predictions. In production, duplicates can arise from:
- Multiple upstream sources sending the same event
- Retry logic creating duplicate entries
- Join operations producing Cartesian products
Detection approaches:
- Exact duplicates: Hash-based deduplication using all or key fields
- Near-duplicates: Similarity measures (edit distance, fuzzy matching) for records that should be unique but have slight variations
- Temporal duplicates: Same entity appearing multiple times within a time window
Referential Integrity
Like database foreign keys, ML data often has relationships. A transaction should reference a valid customer ID; a product review should link to an existing product. Broken references indicate data pipeline failures.
Common checks:
- Foreign key validation against reference tables
- Orphaned records detection
- Consistency of IDs across joined datasets
Cross-Source Consistency
When features come from multiple systems (databases, APIs, data lakes), the same entity might have conflicting values. Monitor for discrepancies in overlapping fields to catch synchronization issues or stale data.
Key Takeaway: Data integrity monitoring detects logical errors—duplicates, broken relationships, and cross-source inconsistencies—that schema and statistical checks miss but can severely degrade model performance and business outcomes.