Course contentsShow
← Machine Learning and Deep Learning
Lesson 2334 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Uncertainty-Aware Models: Ensembles and Probabilistic Dynamics

Using model ensembles or probabilistic models to estimate uncertainty in predictions.

Uncertainty-Aware Models: Ensembles and Probabilistic Dynamics

What you'll learn: How to quantify and leverage model uncertainty in learned dynamics models to make planning safer and more robust.

The Problem with Single Predictions

In the previous lesson, you learned that model errors compound during planning. But there's a deeper issue: a single dynamics model gives you one prediction with no sense of confidence. Imagine asking a friend "What happens if I do X?" and they give you one answer with complete certainty—even when they're guessing. You'd want to know how sure they are!

Model Ensembles: Wisdom of Crowds

The ensemble approach trains multiple independent dynamics models (typically 5-10) with different random initializations or on different data subsets. At prediction time, you:

  • Query all models for the next state prediction
  • Examine the disagreement between predictions
  • Use disagreement as a proxy for uncertainty

When models agree → high confidence. When they disagree → high uncertainty, proceed cautiously.

Probabilistic Dynamics Models

Instead of predicting a single next state, probabilistic models output a distribution (e.g., Gaussian with mean and variance). The model learns both:

  • The expected next state (mean)
  • The uncertainty in that prediction (variance)

This directly captures aleatoric (inherent randomness) and epistemic (knowledge) uncertainty.

Why This Matters for Planning

Uncertainty-aware models enable risk-sensitive planning. Instead of blindly following a plan into uncertain regions where the model might be wrong, your agent can:

  • Prefer actions with predictable outcomes
  • Avoid exploring where model uncertainty is dangerously high
  • Request more data in uncertain regions before committing to risky plans

Key Takeaway: Model ensembles and probabilistic dynamics provide uncertainty estimates that prevent overconfident planning in regions where the learned model may be inaccurate, significantly improving safety and sample efficiency in model-based RL.