Model Error and Compounding Errors in Planning
What You'll Learn
You'll understand why even small prediction errors in learned dynamics models can snowball catastrophically when planning multiple steps ahead.
The Core Problem
When we use a learned model to predict future states, every prediction contains some error. In model-based RL, we use these predictions to plan action sequences. Here's the critical issue: each prediction becomes the input for the next prediction, causing errors to compound exponentially.
Think of it like a GPS with a slight calibration error. After one turn, you're 10 feet off. After ten turns using that offset position, you might be blocks away from where you think you are.
How Errors Accumulate
- Step 1: Model predicts next state with 2% error
- Step 2: Model uses the slightly wrong state to predict again, introducing 2% more error on top of the existing error
- Step 3+: Errors multiply, and the predicted trajectory diverges rapidly from reality
Mathematically, if your model has average error ε per step, after H steps the total error can grow like ε·H² or even exponentially in the worst case.
Planning Horizon Limitations
This compounding effect directly limits how far ahead you can reliably plan:
- Short horizons (1-5 steps): Usually manageable
- Medium horizons (5-20 steps): Errors become noticeable
- Long horizons (20+ steps): Predictions often become useless
This is why model-based methods often combine short-horizon model predictions with model-free value estimates for longer-term planning.
Key Takeaway: Model prediction errors compound multiplicatively over multi-step rollouts, creating an effective planning horizon beyond which learned models become unreliable—a fundamental limitation that model-based RL algorithms must carefully manage.