Course contentsShow
← Machine Learning and Deep Learning
Lesson 2330 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

The Dynamics Model: Predicting Next States and Rewards

Learning transition functions p(s'|s,a) and reward functions to simulate environment behavior.

The Dynamics Model: Predicting Next States and Rewards

What you'll learn: How model-based RL agents learn to predict what happens next in an environment without actually taking actions.

The Core Idea

In model-based reinforcement learning, instead of learning purely from trial and error in the real environment, you build an internal dynamics model that predicts what will happen. This model learns two key functions:

  1. Transition function p(s'|s,a): Given current state s and action a, what's the next state s'?
  2. Reward function r(s,a) or r(s,a,s'): What reward will you receive?

Think of it like learning physics without actually dropping objects: after seeing enough examples, you can predict where a ball will land without throwing it.

Why This Matters

Once you have a dynamics model, you can:

  • Simulate trajectories without interacting with the real (possibly expensive or dangerous) environment
  • Plan ahead by mentally "rolling out" different action sequences
  • Learn more efficiently by generating synthetic experience

The model itself is typically a neural network trained using supervised learning on collected transitions (s, a, s', r).

The Challenge

The dynamics model needs to capture the environment's complexity accurately. For a robot arm, it must learn physics. For a game, it must learn rules and consequences. Poor models lead to model bias: your agent optimizes for a world that doesn't exist, then fails in reality.

Key Takeaway: The dynamics model learns to simulate environment behavior by predicting p(s'|s,a) and rewards, enabling agents to plan and practice in imagination rather than only learning from real experience.