Course contentsShow
← Machine Learning and Deep Learning
Lesson 3030 of 3,53866. Production Monitoring and Data DriftFree lesson

Distance-Based Drift Metrics

Measuring drift with Wasserstein distance, KL divergence, and Population Stability Index for continuous monitoring.

Distance-Based Drift Metrics

What you'll learn: How to quantify data drift using mathematical distance measures that compare distributions in production versus training data.

The Core Idea

While statistical tests (like Kolmogorov-Smirnov) tell you if drift exists, distance-based metrics tell you how much drift you're experiencing. These metrics measure the "distance" between two probability distributions—your training distribution and your production distribution—giving you a continuous score rather than just a yes/no answer.

Think of it like measuring how far you've drifted off course while sailing. A binary alarm tells you "you're off course," but a distance metric tells you "you're 5 nautical miles off course"—much more actionable information.

Three Key Distance Metrics

Wasserstein Distance (Earth Mover's Distance)

Measures the minimum "cost" to transform one distribution into another. Imagine moving piles of dirt: how much effort does it take to reshape your training distribution into your production distribution? Lower values mean similar distributions; it's particularly good for continuous features and robust to outliers.

KL Divergence (Kullback-Leibler)

Measures how much information is lost when you approximate one distribution with another. It's asymmetric (distance from A→B ≠ B→A) and heavily penalizes mismatches in the tails. Works well for probability distributions but can be sensitive to zero probabilities.

Population Stability Index (PSI)

A symmetric metric commonly used in credit scoring and risk models. PSI bucketes data into bins and compares proportions. Values below 0.1 suggest minimal drift, 0.1-0.25 moderate drift, and above 0.25 significant drift requiring investigation.

Why Use These for Continuous Monitoring

These metrics provide numeric severity scores that you can track over time, set gradual alert thresholds on, and correlate with model performance degradation. They turn drift detection from a binary alarm into a monitored vital sign.

Key Takeaway: Distance-based drift metrics (Wasserstein, KL divergence, PSI) quantify how much your production data has drifted, enabling continuous monitoring with numeric thresholds rather than just binary alerts.