Course contentsShow
← Machine Learning and Deep Learning
Lesson 2221 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Experience Replay: Motivation and Mechanics

How storing and resampling past transitions breaks temporal correlations and improves sample efficiency in DQN.

Experience Replay: Motivation and Mechanics

You'll learn: Why DQN stores past experiences in a buffer and randomly samples them during training to break harmful patterns and boost learning efficiency.

Why Do We Need Experience Replay?

In online Q-learning, the agent learns from transitions as they happen in real-time: s, a, r, s'. But there's a critical problem: consecutive experiences are highly correlated. Imagine navigating a maze—your next step is directly influenced by where you just were. When neural networks train on such correlated sequences, they:

  1. Overfit to recent patterns and forget earlier knowledge (catastrophic forgetting)
  2. Exhibit unstable learning because gradients pull the network in rapidly changing directions
  3. Waste valuable experiences by using each transition only once

How Experience Replay Solves This

Think of experience replay like studying for an exam. Instead of reading your textbook once from start to finish (sequential), you collect notes and then shuffle and review them randomly. This randomization helps you generalize better.

The mechanics:

  1. Store transitions (s, a, r, s', done) in a replay buffer (a large memory structure, typically holding 100k–1M transitions)
  2. During training, randomly sample mini-batches from this buffer
  3. Compute DQN loss and update the network using these random samples

Why This Works

  • Breaks temporal correlation: Random sampling mixes experiences from different times and contexts
  • Improves sample efficiency: Each transition is reused multiple times across many updates
  • Stabilizes learning: Diverse batches smooth out noisy gradients

Key Takeaway: Experience replay transforms online RL into a supervised-learning-like problem by storing past transitions and randomly sampling them, breaking correlations and dramatically improving DQN's stability and data efficiency.