Feature Drift vs Covariate Shift
What you'll learn: How to distinguish between changes in individual feature distributions and shifts in the joint input distribution.
The Core Distinction
When monitoring production ML systems, you'll encounter two related but different phenomena. Feature drift refers to changes in individual feature distributions—for example, your user_age feature's mean shifts from 35 to 42 over six months. Covariate shift describes changes in the joint distribution of all input features together—the overall pattern of how features relate to each other changes, even if individual features look stable.
Think of it like ingredients versus recipes. Feature drift is when you start buying sweeter tomatoes (one ingredient changes). Covariate shift is when you start making desserts instead of salads (the combination of ingredients changes entirely).
Why the Distinction Matters
You might observe stable individual features but still experience covariate shift. For instance, age and income distributions could remain unchanged, but the relationship between them shifts—perhaps high-income young users become more common. Your model trained on different age-income patterns will struggle, even though marginal histograms look fine.
Conversely, feature drift in non-critical features might not harm performance if the joint distribution remains stable where it matters for your task.
Monitoring Implications
- Feature drift: Track univariate statistics (mean, variance, quantiles) per feature
- Covariate shift: Monitor multivariate measures like population stability index across feature combinations, or check if your model's learned decision boundaries still align with the data manifold
Both require attention, but covariate shift is often harder to detect because individual feature monitors might miss it entirely.
Key Takeaway: Feature drift tracks individual feature changes; covariate shift captures changes in how features co-occur, requiring multivariate monitoring beyond simple per-feature statistics.