Course contentsShow
← Machine Learning and Deep Learning
Lesson 2333 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Model Error and Compounding Errors in Planning

How prediction errors accumulate over multi-step rollouts and limit planning horizons.

Model Error and Compounding Errors in Planning

What You'll Learn

You'll understand why even small prediction errors in learned dynamics models can snowball catastrophically when planning multiple steps ahead.

The Core Problem

When we use a learned model to predict future states, every prediction contains some error. In model-based RL, we use these predictions to plan action sequences. Here's the critical issue: each prediction becomes the input for the next prediction, causing errors to compound exponentially.

Think of it like a GPS with a slight calibration error. After one turn, you're 10 feet off. After ten turns using that offset position, you might be blocks away from where you think you are.

How Errors Accumulate

  1. Step 1: Model predicts next state with 2% error
  2. Step 2: Model uses the slightly wrong state to predict again, introducing 2% more error on top of the existing error
  3. Step 3+: Errors multiply, and the predicted trajectory diverges rapidly from reality

Mathematically, if your model has average error ε per step, after H steps the total error can grow like ε·H² or even exponentially in the worst case.

Planning Horizon Limitations

This compounding effect directly limits how far ahead you can reliably plan:

  • Short horizons (1-5 steps): Usually manageable
  • Medium horizons (5-20 steps): Errors become noticeable
  • Long horizons (20+ steps): Predictions often become useless

This is why model-based methods often combine short-horizon model predictions with model-free value estimates for longer-term planning.

Key Takeaway: Model prediction errors compound multiplicatively over multi-step rollouts, creating an effective planning horizon beyond which learned models become unreliable—a fundamental limitation that model-based RL algorithms must carefully manage.