Prioritized Experience Replay: Concept
What you'll learn: How to make DQN learn faster by replaying the most important experiences more often.
The Problem with Uniform Sampling
In standard experience replay, you randomly sample transitions from your replay buffer with equal probability. But not all experiences are equally valuable for learning! Some transitions contain surprising information (high TD-error), while others are already well-understood (low TD-error).
Think of studying for an exam: would you spend equal time reviewing every page of the textbook, or focus more on the topics you find challenging?
The Core Idea
Prioritized Experience Replay samples transitions based on their TD-error magnitude. Transitions where the agent's prediction was far from the actual return get sampled more frequently.
The TD-error measures surprise:
TD-error = |reward + γ * max Q(next_state) - Q(state, action)|
Large TD-error means the agent didn't predict this outcome well — that's a learning opportunity!
Why This Accelerates Learning
By prioritizing high-error transitions:
- Rare but important events (like discovering a rare reward or dangerous state) get replayed multiple times instead of being buried in the buffer
- Learning efficiency improves because training focuses on what the agent doesn't understand yet
- Sample efficiency increases — you need fewer environment interactions to reach good performance
You're essentially saying: "Show me my mistakes more often so I can learn from them faster."
The Sampling Strategy
Instead of uniform sampling, each transition gets a priority based on its TD-error. Higher priority = higher sampling probability. This biases learning toward the most informative experiences.
Key Takeaway: Prioritized Experience Replay accelerates learning by sampling transitions proportional to their TD-error, ensuring the agent focuses on surprising or poorly-understood experiences rather than wasting computation on well-learned transitions.