Outlier Detection and Investigation
You'll learn how to systematically identify extreme values using multiple methods and decide whether they're errors, rare events, or valid extremes.
What Are Outliers?
Outliers are data points that sit far away from most others in your dataset—like a 200°F temperature reading in weather data or a person claiming to be 250 years old. But not all outliers are problems! Some represent real rare events (a billionaire's income), while others signal data quality issues (a typo, sensor malfunction, or measurement error).
Multiple Detection Methods
Never rely on just one technique. Use several approaches together:
- Statistical methods: IQR (Interquartile Range) from box plots flags values beyond 1.5×IQR from quartiles
- Z-scores: Values more than 2-3 standard deviations from the mean (assumes normal distribution)
- Visual inspection: Box plots, scatter plots, and histograms reveal extreme points at a glance
- Domain rules: Values outside physically possible ranges (negative age, 500% growth rate)
Investigation Process
Once detected, investigate each outlier:
- Verify the value against source data—is it a recording error?
- Check context—does it appear in a cluster of suspicious records?
- Apply domain knowledge—could this happen in reality?
- Document your decision—keep or remove? Transform or flag?
Think of outlier investigation like airport security: automated scanners flag suspicious bags, but trained experts must examine each one to decide if it's dangerous or just unusual.
Key Takeaway: Outlier detection requires multiple methods working together, and every suspicious value deserves investigation using both statistical evidence and domain expertise before deciding its fate.