Course contentsShow
← Machine Learning and Deep Learning
Lesson 2332 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Model Learning Objectives and Supervised Training

Framing dynamics learning as supervised prediction using collected trajectories as training data.

Model Learning Objectives and Supervised Training

What you'll learn: How to frame the problem of learning environment dynamics as a standard supervised learning task using collected agent experience.

The Core Idea

In model-based RL, you need to learn how the environment works—specifically, how states transition and what rewards appear. The elegant insight is that this is just supervised learning in disguise. Every trajectory your agent collects contains training examples: given state s and action a, the environment "labels" your data with the next state s' and reward r. You're predicting outcomes, not discovering patterns in unlabeled data.

Setting Up the Learning Problem

Think of your dynamics model as a student learning physics by watching experiments. Each transition (s, a, s', r) from your replay buffer becomes a supervised example:

  • Input (X): current state s and action a
  • Target (y): next state s' and/or reward r

Your model (typically a neural network) learns the mapping: f(s, a) → (s', r).

The Training Process

You collect trajectories through environment interaction, store them in a replay buffer, then sample mini-batches to train your dynamics model using standard supervised losses:

  • For next state: Mean squared error (MSE) between predicted ŝ' and actual s'
  • For reward: MSE or cross-entropy depending on whether rewards are continuous or discrete

This is identical to training any regression or classification model—backpropagation, gradient descent, the works. The only twist: your training data keeps growing as the agent explores.

Key Takeaway: Learning environment dynamics reduces to supervised prediction where your agent's own experience provides labeled training data—every interaction is a teacher signal for how the world behaves.