Course contentsShow
← Machine Learning and Deep Learning
Lesson 3029 of 3,53866. Production Monitoring and Data DriftFree lesson

Statistical Tests for Drift Detection

Using Kolmogorov-Smirnov, Chi-squared, and other statistical tests to detect distribution changes in features.

Statistical Tests for Drift Detection

What you'll learn: How to use formal statistical tests to detect when your input data distributions have changed in production.

The Need for Rigorous Detection

You know data drift happens—features shift, user behavior changes, external conditions evolve. But gut feelings aren't enough. You need quantitative, reproducible ways to detect these changes before they silently degrade your model's performance.

Statistical Tests as Drift Detectors

Think of statistical tests as mathematical "smoke detectors" for your data. They compare a reference distribution (from training or a baseline period) against a current window of production data, then answer: "How likely is it that these samples came from the same distribution?"

Kolmogorov-Smirnov (KS) Test

Best for: Continuous numerical features

The KS test measures the maximum distance between two cumulative distribution functions (CDFs). If production data's CDF has drifted far from your reference, the test yields a small p-value, signaling drift.

Intuition: Imagine plotting what percentage of users are under each age value. If production's curve suddenly shifts right (aging users), KS catches that displacement.

Chi-Squared Test

Best for: Categorical features

This test compares observed category frequencies against expected ones. If your "country" feature suddenly shows 40% more traffic from a new region, chi-squared detects that proportional shift.

Intuition: You expected 1000 clicks per category; you got 800 and 1200. Chi-squared quantifies if that's random noise or real drift.

Other Common Tests

  • Wasserstein Distance: Measures "effort" to transform one distribution into another
  • Population Stability Index (PSI): Bins data and compares distributions via log ratios
  • Jensen-Shannon Divergence: Symmetric measure of distribution similarity

Practical Considerations

Set appropriate significance levels (α = 0.05 is common but context-dependent). Test frequently enough to catch drift early, but not so often you get false alarms from random fluctuations.

Key Takeaway: Statistical tests like Kolmogorov-Smirnov and Chi-squared provide objective, automated ways to detect feature distribution drift by comparing production data against reference distributions with mathematically rigorous thresholds.