Deterministic Policy Gradients
What You'll Learn
You'll understand why deterministic policies can be more sample-efficient than stochastic ones, and how the deterministic policy gradient theorem enables this.
The Core Idea
So far, you've worked with stochastic policies π(a|s) that output probability distributions over actions. But what if your policy directly outputs a single action: a = μ(s)? This is a deterministic policy.
The breakthrough insight: instead of sampling actions and computing gradients through the log-probability trick (score function estimator), you can compute gradients directly through the action-value function Q(s, a).
The Deterministic Policy Gradient Theorem
The deterministic policy gradient has an elegant form:
∇_θ J(θ) = E_s[ ∇_θ μ_θ(s) · ∇_a Q(s,a)|_{a=μ_θ(s)} ]
In plain English: the gradient of performance is the expected gradient of your policy multiplied by how Q changes with respect to the action.
Why is this more sample-efficient?
- No need to sample multiple actions — you only need one action per state
- No integration over action space — stochastic policies must integrate ∇π·Q over all possible actions
- Lower variance gradients — deterministic gradients eliminate the randomness from action sampling
Think of it like this: instead of rolling dice repeatedly to estimate which direction improves things (stochastic), you follow the compass needle directly (deterministic).
When to Use Deterministic Policies
Deterministic policies shine in continuous action spaces where exploring every possible action is impractical. For discrete actions, stochastic policies often remain preferable for exploration.
Key Takeaway: Deterministic policy gradients compute gradients directly through Q-functions without action sampling, making them more sample-efficient than stochastic alternatives—especially valuable in continuous control tasks.