Forecast Evaluation Metrics
What you'll learn: How to measure how good your time series forecasts are using numerical metrics and proper validation techniques.
Why Measuring Forecast Accuracy Matters
After building a time series model like SARIMA or Holt-Winters, you need to know: how well does it actually predict the future? Just like you wouldn't judge a basketball player without counting their shots made, you can't evaluate forecasts without quantitative metrics.
Common Forecast Error Metrics
Think of forecast errors as the gap between what you predicted and what actually happened. Here are the most popular ways to measure that gap:
Mean Absolute Error (MAE)
The average of absolute errors. If you predicted [100, 200, 300] and actual values were [110, 190, 320], your errors are [10, 10, 20], so MAE = 13.33. Easy to interpret — it's in the same units as your data.
Root Mean Squared Error (RMSE)
Square the errors, average them, then take the square root. Penalizes large errors more heavily than MAE. Useful when big mistakes are particularly costly.
Mean Absolute Percentage Error (MAPE)
Expresses errors as percentages: if actual = 100 and forecast = 110, that's a 10% error. Great for comparing across different scales, but fails when actual values are near zero.
Hold-Out Validation for Time Series
Unlike regular machine learning, you cannot randomly split time series data — that breaks the time order! Instead, use the most recent data as your test set. Train on Jan–Oct, test on Nov–Dec. This mimics real forecasting: predicting truly unseen future periods.
Key Takeaway: Use MAE, RMSE, and MAPE to quantify forecast accuracy, and always validate on chronologically held-out data to ensure your model can truly predict the future.