Course contentsShow
← Machine Learning and Deep Learning
Lesson 2227 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Prioritized Experience Replay: Concept

Sampling important transitions more frequently based on TD-error to accelerate learning from rare events.

Prioritized Experience Replay: Concept

What you'll learn: How to make DQN learn faster by replaying the most important experiences more often.

The Problem with Uniform Sampling

In standard experience replay, you randomly sample transitions from your replay buffer with equal probability. But not all experiences are equally valuable for learning! Some transitions contain surprising information (high TD-error), while others are already well-understood (low TD-error).

Think of studying for an exam: would you spend equal time reviewing every page of the textbook, or focus more on the topics you find challenging?

The Core Idea

Prioritized Experience Replay samples transitions based on their TD-error magnitude. Transitions where the agent's prediction was far from the actual return get sampled more frequently.

The TD-error measures surprise:

TD-error = |reward + γ * max Q(next_state) - Q(state, action)|

Large TD-error means the agent didn't predict this outcome well — that's a learning opportunity!

Why This Accelerates Learning

By prioritizing high-error transitions:

  • Rare but important events (like discovering a rare reward or dangerous state) get replayed multiple times instead of being buried in the buffer
  • Learning efficiency improves because training focuses on what the agent doesn't understand yet
  • Sample efficiency increases — you need fewer environment interactions to reach good performance

You're essentially saying: "Show me my mistakes more often so I can learn from them faster."

The Sampling Strategy

Instead of uniform sampling, each transition gets a priority based on its TD-error. Higher priority = higher sampling probability. This biases learning toward the most informative experiences.

Key Takeaway: Prioritized Experience Replay accelerates learning by sampling transitions proportional to their TD-error, ensuring the agent focuses on surprising or poorly-understood experiences rather than wasting computation on well-learned transitions.