Uncertainty-Aware Models: Ensembles and Probabilistic Dynamics
What you'll learn: How to quantify and leverage model uncertainty in learned dynamics models to make planning safer and more robust.
The Problem with Single Predictions
In the previous lesson, you learned that model errors compound during planning. But there's a deeper issue: a single dynamics model gives you one prediction with no sense of confidence. Imagine asking a friend "What happens if I do X?" and they give you one answer with complete certainty—even when they're guessing. You'd want to know how sure they are!
Model Ensembles: Wisdom of Crowds
The ensemble approach trains multiple independent dynamics models (typically 5-10) with different random initializations or on different data subsets. At prediction time, you:
- Query all models for the next state prediction
- Examine the disagreement between predictions
- Use disagreement as a proxy for uncertainty
When models agree → high confidence. When they disagree → high uncertainty, proceed cautiously.
Probabilistic Dynamics Models
Instead of predicting a single next state, probabilistic models output a distribution (e.g., Gaussian with mean and variance). The model learns both:
- The expected next state (mean)
- The uncertainty in that prediction (variance)
This directly captures aleatoric (inherent randomness) and epistemic (knowledge) uncertainty.
Why This Matters for Planning
Uncertainty-aware models enable risk-sensitive planning. Instead of blindly following a plan into uncertain regions where the model might be wrong, your agent can:
- Prefer actions with predictable outcomes
- Avoid exploring where model uncertainty is dangerously high
- Request more data in uncertain regions before committing to risky plans
Key Takeaway: Model ensembles and probabilistic dynamics provide uncertainty estimates that prevent overconfident planning in regions where the learned model may be inaccurate, significantly improving safety and sample efficiency in model-based RL.