Completeness Checks
What you'll learn: How to systematically detect and measure missing, null, or incomplete data across your datasets to understand data quality.
Why Completeness Matters
Imagine receiving a survey where half the participants skipped the age question—your analysis about age groups would be incomplete or misleading. Completeness checks are your detective work for finding exactly where data is missing, how much is missing, and whether the missingness follows patterns.
You've already validated data types (schema) and ranges (constraints). Now you're asking: "Is the data actually there?"
Core Components of Completeness Checks
Identifying Missing Values
Missing data appears in many forms: NaN, None, empty strings, placeholder values like -999, or simply absent rows. Your first task is detecting these across all columns.
import pandas as pd
# Count missing values per column
missing_counts = df.isnull().sum()
# Calculate percentage missing
missing_pct = (df.isnull().sum() / len(df)) * 100
Quantifying the Problem
Don't just count nulls—measure their impact. A column with 95% missing data requires different handling than one with 2% missing.
Finding Null Patterns
Sometimes missingness isn't random. Perhaps all records from a specific source are missing a field, or certain combinations always have gaps. Checking correlations between missing values reveals these patterns.
# Check if missing values occur together
import missingno as msno
msno.matrix(df) # Visual pattern detection
msno.heatmap(df) # Correlation of missingness
When to Check Completeness
Run completeness checks immediately after loading data and before any analysis. Missing data can bias results, crash models, or produce silently incorrect calculations.
Key Takeaway: Completeness checks quantify what's missing in your dataset—counting nulls, calculating percentages, and identifying patterns helps you decide whether to drop, fill, or flag incomplete records before analysis.