When to Use Model-Based RL: Sample Efficiency Trade-offs
What you'll learn: You'll understand the specific scenarios where model-based RL shines—particularly when data is scarce or you're transferring from simulation to real-world systems.
The Core Trade-off
Model-based RL learns a dynamics model of the environment, then uses it for planning. This adds complexity but can dramatically reduce the number of real environment interactions needed. The key question is: when is this trade-off worth it?
Think of it like learning to drive. Model-free methods are like practicing only in a real car—expensive and risky, but you learn the true dynamics. Model-based methods are like practicing in a simulator first—you can make thousands of mistakes cheaply, though the simulator isn't perfect.
Data-Limited Domains
Model-based methods excel when real-world interactions are expensive, dangerous, or slow:
- Robotics: Physical wear, safety concerns, and real-time constraints make each episode costly
- Healthcare: Patient trials are limited and high-stakes
- Industrial processes: Chemical plants or manufacturing lines can't be reset thousands of times
- Financial trading: Real capital is at risk
In these domains, learning a model from limited data, then planning extensively in that model, beats collecting millions of real samples.
Sim-to-Real Transfer
Model-based RL bridges the simulation-to-reality gap:
- Train a dynamics model in simulation (abundant data)
- Fine-tune with minimal real-world data
- Use uncertainty-aware models to identify where sim and reality diverge
- Perform robust planning that accounts for model error
This leverages cheap simulation data while adapting to real-world dynamics with just hundreds—not millions—of real samples.
When Model-Free Wins
If you have unlimited environment access (video games, simple simulations), model-free methods like PPO or SAC are often simpler and more robust to model errors.
Key Takeaway: Use model-based RL when environment samples are expensive or limited, especially in robotics and sim-to-real scenarios where sample efficiency matters more than asymptotic performance.