Model Learning Objectives and Supervised Training
What you'll learn: How to frame the problem of learning environment dynamics as a standard supervised learning task using collected agent experience.
The Core Idea
In model-based RL, you need to learn how the environment works—specifically, how states transition and what rewards appear. The elegant insight is that this is just supervised learning in disguise. Every trajectory your agent collects contains training examples: given state s and action a, the environment "labels" your data with the next state s' and reward r. You're predicting outcomes, not discovering patterns in unlabeled data.
Setting Up the Learning Problem
Think of your dynamics model as a student learning physics by watching experiments. Each transition (s, a, s', r) from your replay buffer becomes a supervised example:
- Input (X): current state
sand actiona - Target (y): next state
s'and/or rewardr
Your model (typically a neural network) learns the mapping: f(s, a) → (s', r).
The Training Process
You collect trajectories through environment interaction, store them in a replay buffer, then sample mini-batches to train your dynamics model using standard supervised losses:
- For next state: Mean squared error (MSE) between predicted
ŝ'and actuals' - For reward: MSE or cross-entropy depending on whether rewards are continuous or discrete
This is identical to training any regression or classification model—backpropagation, gradient descent, the works. The only twist: your training data keeps growing as the agent explores.
Key Takeaway: Learning environment dynamics reduces to supervised prediction where your agent's own experience provides labeled training data—every interaction is a teacher signal for how the world behaves.