World Models and Latent Imagination
What You'll Learn
You'll discover how agents can learn compact, compressed representations of environment dynamics in a latent space, enabling efficient planning without directly processing high-dimensional observations like images.
The Core Idea
Imagine trying to plan your route through a city. You don't mentally replay every pixel of every street view—instead, you think with a compressed "mental map" that captures the essential structure. World models do the same for RL agents.
Traditional model-based RL learns dynamics in the raw observation space (e.g., predicting the next 84×84×4 Atari frame). This is computationally expensive and often contains irrelevant details. World models instead learn to:
- Encode high-dimensional observations into a low-dimensional latent representation (the compressed "essence" of the state)
- Learn dynamics in this latent space (predicting the next latent state given actions)
- Decode back to observations when needed
- Plan entirely in the efficient latent space
Latent Imagination
Latent imagination is the process of planning by "imagining" future trajectories in latent space. The agent:
- Encodes the current observation into a latent state
- Simulates forward using the learned latent dynamics model
- Evaluates imagined trajectories using a value function
- Selects actions based on these mental simulations
This is dramatically more sample-efficient than learning purely from real environment interactions, because the agent can practice thousands of imagined rollouts quickly.
Why It Matters
By compressing a 10,000-dimensional pixel observation into perhaps a 32-dimensional latent vector, planning becomes 100× faster. The latent space naturally filters out irrelevant details (like background textures) and focuses on controllable, predictive features.
Key Takeaway: World models with latent imagination compress environment dynamics into low-dimensional representations, enabling agents to plan efficiently by "imagining" trajectories in a learned latent space rather than expensive pixel space.