Course contentsShow
← Machine Learning and Deep Learning
Lesson 2265 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

The Policy Gradient Theorem

Mathematical foundation showing how to compute gradients of expected return with respect to policy parameters.

The Policy Gradient Theorem

What You'll Learn

You'll understand the mathematical foundation that allows us to compute gradients of expected return with respect to policy parameters — the core result that makes policy optimization possible.

The Core Challenge

In policy-based methods, we want to adjust our policy parameters θ to maximize expected return. But how do we compute ∇_θ J(θ), the gradient of performance with respect to parameters, when the policy influences not just immediate rewards but the entire distribution of state visits?

The Theorem's Insight

The Policy Gradient Theorem provides an elegant solution. It shows that we can compute the gradient without needing to differentiate through the environment dynamics (which we often don't know).

The theorem states:

∇_θ J(θ) ∝ E[∇_θ log π_θ(a|s) · Q^π(s,a)]

In plain English: The gradient of expected return equals the expected gradient of the log-probability of actions, weighted by how good those actions are (the Q-value).

Why This Matters

This formula is remarkable because:

  • No environment model needed: We don't differentiate through state transitions
  • Sample-based estimation: We can estimate this expectation from experience
  • Action weighting: Good actions (high Q-value) get pushed up; bad actions get pushed down

Think of it like adjusting a bias: when you take an action that leads to high return, you increase the probability of that action in that state. When it leads to low return, you decrease it.

The Building Block

This theorem underpins virtually all modern policy gradient algorithms. It transforms the abstract problem of "improve the policy" into a concrete gradient we can compute and use with standard optimization methods.

Key Takeaway: The Policy Gradient Theorem gives us a tractable way to compute policy gradients by weighting log-probability gradients with action values, enabling direct policy optimization without environment models.