Mean Absolute Error and Huber Loss
What you'll learn: How MAE and Huber loss provide more robust alternatives to MSE when your data contains outliers.
The Problem with MSE's Squaring
You've learned that Mean Squared Error (MSE) penalizes errors quadratically. While this works well for clean data, it becomes extremely sensitive to outliers. A single data point far from the prediction can dominate the loss because squaring amplifies large errors dramatically.
Mean Absolute Error (MAE)
Mean Absolute Error takes the absolute value of errors instead of squaring them:
MAE = (1/n) × Σ|y_actual - y_predicted|
Analogy: If MSE is like penalizing someone exponentially more for being farther off target, MAE treats all distances proportionally. Being twice as far off costs exactly twice as much.
Key advantages:
- More robust to outliers (large errors don't explode)
- Easier to interpret (same units as your target variable)
Trade-off: MAE's gradient is constant, which can make optimization less smooth near the minimum.
Huber Loss: The Best of Both Worlds
Huber loss is a hybrid that acts like MSE for small errors (smooth gradients) and like MAE for large errors (outlier protection):
- For errors smaller than threshold δ: behaves like MSE
- For errors larger than δ: behaves like MAE
This gives you smooth, fast convergence while protecting against outliers that would otherwise dominate training.
When to use what:
- MSE: Clean data, you want to heavily penalize large errors
- MAE: Outliers present, you want equal treatment of all errors
- Huber: Best general-purpose choice when you're unsure about outliers
Key Takeaway: MAE provides outlier-robust regression through linear error penalization, while Huber loss combines MSE's smooth optimization with MAE's outlier resilience by switching behavior based on error magnitude.