Course contentsShow
← Machine Learning and Deep Learning
Lesson 2336 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

When to Use Model-Based RL: Sample Efficiency Trade-offs

Understanding scenarios where model-based methods excel: data-limited domains and sim-to-real transfer.

When to Use Model-Based RL: Sample Efficiency Trade-offs

What you'll learn: You'll understand the specific scenarios where model-based RL shines—particularly when data is scarce or you're transferring from simulation to real-world systems.

The Core Trade-off

Model-based RL learns a dynamics model of the environment, then uses it for planning. This adds complexity but can dramatically reduce the number of real environment interactions needed. The key question is: when is this trade-off worth it?

Think of it like learning to drive. Model-free methods are like practicing only in a real car—expensive and risky, but you learn the true dynamics. Model-based methods are like practicing in a simulator first—you can make thousands of mistakes cheaply, though the simulator isn't perfect.

Data-Limited Domains

Model-based methods excel when real-world interactions are expensive, dangerous, or slow:

  • Robotics: Physical wear, safety concerns, and real-time constraints make each episode costly
  • Healthcare: Patient trials are limited and high-stakes
  • Industrial processes: Chemical plants or manufacturing lines can't be reset thousands of times
  • Financial trading: Real capital is at risk

In these domains, learning a model from limited data, then planning extensively in that model, beats collecting millions of real samples.

Sim-to-Real Transfer

Model-based RL bridges the simulation-to-reality gap:

  1. Train a dynamics model in simulation (abundant data)
  2. Fine-tune with minimal real-world data
  3. Use uncertainty-aware models to identify where sim and reality diverge
  4. Perform robust planning that accounts for model error

This leverages cheap simulation data while adapting to real-world dynamics with just hundreds—not millions—of real samples.

When Model-Free Wins

If you have unlimited environment access (video games, simple simulations), model-free methods like PPO or SAC are often simpler and more robust to model errors.

Key Takeaway: Use model-based RL when environment samples are expensive or limited, especially in robotics and sim-to-real scenarios where sample efficiency matters more than asymptotic performance.