Course contentsShow
← Machine Learning and Deep Learning
Lesson 2335 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Model Predictive Control with Learned Models

Planning actions by optimizing over short horizons using the learned dynamics model.

Model Predictive Control with Learned Models

What You'll Learn

You'll learn how to plan optimal actions over short time horizons using the dynamics model you've learned, rather than relying solely on value functions or policies.

The Core Idea

Model Predictive Control (MPC) is a planning strategy where you use your learned dynamics model to simulate future trajectories, evaluate them, and pick the best action sequence—but you only execute the first action, then re-plan.

Think of it like driving: instead of memorizing every turn, you continuously look ahead a few seconds, plan your steering and speed, make the first move, then look ahead again with updated information.

How MPC Works with Learned Models

  1. Starting from current state s_t, use your learned model to predict what happens if you take different action sequences
  2. Roll out multiple candidate trajectories over a short horizon (e.g., 10-20 steps)
  3. Score each trajectory by summing predicted rewards
  4. Select the best first action from the highest-scoring sequence
  5. Execute only that first action, observe the real next state
  6. Repeat from the new state

This is called "receding horizon" control because you keep moving your planning window forward.

Why This Matters

MPC leverages your learned dynamics model (potentially an ensemble for uncertainty) without needing to train a separate policy network. It adapts instantly to changes because you re-plan at every step. The short horizon limits how much model error compounds—you're only predicting a few steps ahead before correcting with real observations.

The optimization over action sequences can use various methods: random shooting, cross-entropy method (CEM), or gradient-based optimization if your model is differentiable.

Key Takeaway: Model Predictive Control uses your learned dynamics model to repeatedly plan short action sequences, executing only the first action before re-planning—balancing model-based foresight with real-world feedback to limit error accumulation.