Statistical Tests for Drift Detection
What you'll learn: How to use formal statistical tests to detect when your input data distributions have changed in production.
The Need for Rigorous Detection
You know data drift happens—features shift, user behavior changes, external conditions evolve. But gut feelings aren't enough. You need quantitative, reproducible ways to detect these changes before they silently degrade your model's performance.
Statistical Tests as Drift Detectors
Think of statistical tests as mathematical "smoke detectors" for your data. They compare a reference distribution (from training or a baseline period) against a current window of production data, then answer: "How likely is it that these samples came from the same distribution?"
Kolmogorov-Smirnov (KS) Test
Best for: Continuous numerical features
The KS test measures the maximum distance between two cumulative distribution functions (CDFs). If production data's CDF has drifted far from your reference, the test yields a small p-value, signaling drift.
Intuition: Imagine plotting what percentage of users are under each age value. If production's curve suddenly shifts right (aging users), KS catches that displacement.
Chi-Squared Test
Best for: Categorical features
This test compares observed category frequencies against expected ones. If your "country" feature suddenly shows 40% more traffic from a new region, chi-squared detects that proportional shift.
Intuition: You expected 1000 clicks per category; you got 800 and 1200. Chi-squared quantifies if that's random noise or real drift.
Other Common Tests
- Wasserstein Distance: Measures "effort" to transform one distribution into another
- Population Stability Index (PSI): Bins data and compares distributions via log ratios
- Jensen-Shannon Divergence: Symmetric measure of distribution similarity
Practical Considerations
Set appropriate significance levels (α = 0.05 is common but context-dependent). Test frequently enough to catch drift early, but not so often you get false alarms from random fluctuations.
Key Takeaway: Statistical tests like Kolmogorov-Smirnov and Chi-squared provide objective, automated ways to detect feature distribution drift by comparing production data against reference distributions with mathematically rigorous thresholds.