Model-Based vs Model-Free RL: The Fundamental Distinction
You'll learn: The core difference between learning a predictive model of your environment versus learning how to act directly.
The Core Distinction
In reinforcement learning, there are two fundamentally different approaches to solving problems:
Model-Free RL learns policies or value functions directly from experience, without trying to understand how the environment works. All the methods you've studied so far—DQN, PPO, SAC, TD3—are model-free. They learn "what to do" without learning "what happens next."
Model-Based RL learns a model of the environment's dynamics: given a state and action, what will the next state and reward be? With this predictive model, the agent can plan ahead, simulate different action sequences, and choose the best path forward.
The Planning vs Learning Trade-off
Think of learning to play chess:
-
Model-Free: You play thousands of games, slowly learning which moves lead to wins. You don't explicitly learn the rules—you just develop intuition about good moves.
-
Model-Based: You first learn the rules (how pieces move, what leads to checkmate). Then you can mentally simulate different move sequences before deciding what to do.
Model-based methods can be more sample-efficient because they can "imagine" outcomes without actually experiencing them. However, they face a challenge: if your model of the environment is wrong, your entire planning process may be flawed.
Model-free methods like PPO and SAC sidestep this by learning directly from real experience, though they typically need more interactions with the environment.
Key Takeaway: Model-free RL learns what to do directly from experience; model-based RL learns how the world works first, then uses that model to plan optimal actions.