The ESD Component
What you'll learn: How the Extreme Studentized Deviate (ESD) test handles multiple outliers simultaneously without letting them hide each other.
The Core Challenge
When you apply traditional outlier tests like Grubbs' Test sequentially (as you learned in lesson 1396), you face two serious problems:
- Masking: One outlier hides another because it shifts the mean and standard deviation
- Swamping: A valid point gets falsely flagged because outliers distort the statistics
The ESD component solves both problems by testing for multiple outliers at once.
How ESD Works
The Generalized Extreme Studentized Deviate (GESD) test runs iteratively but intelligently:
- You specify the maximum number of suspected outliers (let's call it k)
- ESD removes one outlier at a time (like Grubbs'), computing a test statistic for each removal
- After k iterations, it examines all test statistics together against dynamically calculated critical values
- It identifies the optimal stopping point—catching all real outliers without false positives
Think of it like peeling an onion: you remove layers (outliers) one by one, but you don't decide which layers were bad until you've examined the whole sequence. This prevents early removals from fooling later decisions.
Why It's Essential in Seasonal Hybrid ESD
In time series anomaly detection (like the Seasonal Hybrid ESD you just learned about), after removing seasonality and trend, you're left with residuals that may contain multiple anomalies. Using ESD on these residuals ensures you catch clustered anomalies or outlier storms without statistical distortion.
Key Takeaway: The ESD component detects multiple outliers simultaneously by evaluating a sequence of removal tests together, eliminating the masking and swamping effects that plague sequential single-outlier tests.