Course contentsShow
← Data Science
Lesson 790 of 2,14520. Time Series ModelsFree lesson

Out-of-Sample Forecast Evaluation

Assessing model performance on held-out data using metrics like MAE, RMSE, and MAPE for forecast accuracy.

Out-of-Sample Forecast Evaluation

What you'll learn: How to assess your time series model's real-world forecasting ability by testing it on data it has never seen before.

Why We Need Out-of-Sample Testing

Imagine training a weather forecaster using only last year's data, then asking them to predict tomorrow's weather. How well they perform on tomorrow (data they haven't seen) matters far more than how well they "predicted" last year (data they already knew).

In time series forecasting, out-of-sample evaluation means holding back recent data points, building your model on the earlier data, then measuring how accurately it predicts the held-out future. This tests whether your model truly captures patterns that generalize forward in time, not just fits past noise.

The Process

  1. Split your data: Reserve the last portion (e.g., last 20% or last 12 months) as your test set
  2. Train on the rest: Build your ARIMA, Holt-Winters, or other model using only the training portion
  3. Forecast ahead: Generate predictions for the held-out period
  4. Calculate error metrics: Compare predictions to actual values

Key Metrics for Forecast Accuracy

  • MAE (Mean Absolute Error): Average of absolute differences; easy to interpret in original units
  • RMSE (Root Mean Squared Error): Penalizes larger errors more heavily; sensitive to outliers
  • MAPE (Mean Absolute Percentage Error): Expresses error as a percentage; useful for comparing across scales

Lower values mean better forecasts. Choose metrics based on your problem—MAPE works well when relative errors matter, while MAE/RMSE suit scenarios where absolute prediction errors are critical.

Key Takeaway: Out-of-sample forecast evaluation using metrics like MAE, RMSE, and MAPE reveals whether your time series model can truly predict the future, not just explain the past.