Outliers in Regression
What you'll learn: You'll discover how to distinguish between three types of unusual observations in regression—outliers, high-leverage points, and influential points—using residual diagnostics.
Understanding the Three Types of Unusual Points
Not all unusual observations affect your regression model the same way. Think of regression like a seesaw balanced on a fulcrum:
Outliers are data points with unusually large residuals—they lie far from the fitted regression line vertically. These points don't fit the overall pattern well, but they may not significantly change the regression line itself if they're not positioned at extreme predictor values.
High-leverage points have extreme values of the predictor variable(s). They sit far from the mean of X, like someone standing at the very edge of a seesaw. These points could potentially pull the regression line toward themselves, but they don't always do so—they have the potential for influence.
Influential points actually do pull the regression line significantly. If you removed an influential point and refitted the model, your slope or intercept would change noticeably. A point can be influential because it's both an outlier and has high leverage, or sometimes just one or the other in extreme cases.
Using Diagnostics to Distinguish Them
- Standardized or studentized residuals help identify outliers (typically |residual| > 2 or 3 suggests an outlier)
- Leverage values identify high-leverage points (points with unusually extreme X values)
- Cook's Distance measures actual influence—it combines residual size and leverage to show which points change the regression line most
A point might have high leverage without being influential if it actually follows the trend. Conversely, a point might be an outlier but not influential if it's near the center of the X range.
Key Takeaway: Outliers have large residuals, high-leverage points have extreme predictor values, and influential points actually change the regression line—use residual plots, leverage, and Cook's Distance to distinguish them.