Course contentsShow
← Machine Learning and Deep Learning
Lesson 2214 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Frame Stacking and State Representation

Why raw frames lack temporal information and how stacking consecutive frames creates Markovian states.

Frame Stacking and State Representation

What you'll learn: How to give your DQN agent a sense of motion by combining multiple frames into a single state representation.

The Problem with Single Frames

A single raw frame from a video game or visual environment is like a photograph—it shows you where things are, but not how they're moving. Imagine trying to play Pong with just one frozen image: you'd see the ball's position but have no idea if it's moving left, right, up, or down. This violates the Markov Property we rely on—the current state doesn't contain enough information to predict future states accurately.

Without velocity information, your agent can't distinguish between a ball moving toward the paddle versus away from it, even though these situations demand completely different actions.

Frame Stacking: Creating Temporal Context

Frame stacking solves this by concatenating the last k consecutive frames (typically 4) into a single state representation. Now your state isn't just one image—it's a short "video clip" that captures motion through differences between frames.

When you stack 4 grayscale frames of size 84×84, you create a 84×84×4 tensor. The neural network can now "see" trajectories: if the ball appears in slightly different positions across the stacked frames, the network learns the direction and speed of movement.

This transformed representation approximates a Markovian state because velocity (and acceleration) are now implicitly encoded through temporal differences.

Why This Works

Think of frame stacking like a flipbook animation—one page shows a static drawing, but flip through several pages quickly and you perceive motion. Your DQN's convolutional layers process these stacked frames to extract motion patterns, just as your brain does.

Key Takeaway: Frame stacking converts non-Markovian single frames into approximately Markovian states by encoding temporal information through consecutive frame history, enabling value-based methods to learn from visual observations.