Deep Deterministic Policy Gradient (DDPG)
What you'll learn: How DDPG combines deterministic policy gradients with Q-learning techniques to create a powerful actor-critic algorithm for continuous control tasks.
The Core Idea
DDPG is a model-free, off-policy actor-critic algorithm designed specifically for continuous action spaces. Think of it as "DQN meets deterministic policy gradients." Just as DQN learned which discrete action had the highest Q-value, DDPG learns a deterministic policy that directly outputs the best continuous action—but it uses Q-learning machinery to do so.
Actor-Critic Architecture for Continuous Control
DDPG maintains four networks:
- Actor network μ(s|θ): Takes a state and outputs a deterministic action (not a probability distribution)
- Critic network Q(s,a|w): Evaluates state-action pairs, just like in DQN
- Target actor and target critic: Slowly-updated copies for stability (borrowed from DQN's target network idea)
The critic learns using temporal difference updates, similar to DQN, estimating Q-values for state-action pairs. The actor learns by climbing the Q-value gradient—it adjusts its parameters to output actions that the critic believes are better.
Why This Combination Works
In continuous spaces, you can't enumerate all actions to find the max Q-value. DDPG's deterministic actor solves this: it directly computes the action that maximizes Q. The critic provides the training signal by evaluating how good those actions are. Experience replay (from DQN) and target networks stabilize learning, while the deterministic policy gradient theorem ensures the actor improves.
Key Takeaway: DDPG extends DQN's stability tricks (replay buffer, target networks) to continuous control by pairing a deterministic actor that outputs actions with a critic that evaluates them, creating an efficient off-policy algorithm for robotics and control tasks.