Course contentsShow
← Machine Learning and Deep Learning
Lesson 2319 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

DDPG: Experience Replay and Target Networks

Using replay buffers and target networks in DDPG to stabilize off-policy learning in continuous spaces.

DDPG: Experience Replay and Target Networks

What you'll learn: How DDPG borrows DQN's stabilization tricks—experience replay and target networks—to make off-policy learning work in continuous action spaces.

The Stability Challenge in DDPG

DDPG combines deterministic policy gradients with deep neural networks for both actor and critic. But training two neural networks that depend on each other's outputs is inherently unstable—like trying to balance on a seesaw while someone else adjusts their position based on yours.

Experience Replay: Breaking Temporal Correlations

Just as in DQN, DDPG stores transitions (s, a, r, s', done) in a replay buffer. During training, the algorithm samples random minibatches from this buffer rather than learning from consecutive experiences.

Why this matters: Consecutive timesteps in continuous control are highly correlated (similar states, similar actions). Learning from them directly causes the networks to overfit to recent patterns and forget past lessons. Random sampling breaks these correlations, making gradient updates more stable and data-efficient.

Target Networks: Slowing Down the Moving Goal

DDPG maintains four networks total:

  • Actor network (current policy)
  • Actor target network (slowly updated copy)
  • Critic network (current Q-function)
  • Critic target network (slowly updated copy)

The target networks compute the bootstrap targets for the critic's TD error, providing a stable reference point. They update via soft updates (Polyak averaging):

θ_target ← τ·θ + (1-τ)·θ_target

where τ is typically very small (0.001–0.005), meaning targets change gradually.

The analogy: Imagine learning to hit a moving target. If the target jumps around frantically (regular networks), you'll never converge. Target networks make it drift slowly, giving you time to adjust your aim.

Key Components Working Together

Experience replay provides diverse, uncorrelated data. Target networks provide stable learning targets. Together, they transform DDPG from an unstable algorithm into one that can learn complex continuous control tasks.

Key Takeaway: DDPG stabilizes off-policy learning in continuous spaces by combining experience replay (for decorrelated training data) with slowly-updated target networks (for stable Q-value targets)—directly importing DQN's most important innovations.