Course contentsShow
← Machine Learning and Deep Learning
Lesson 2250 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

The Policy Gradient Theorem

Formal derivation showing how to compute gradients of expected return with respect to policy parameters.

The Policy Gradient Theorem

What you'll learn: How to mathematically derive gradients of expected return with respect to policy parameters—the foundation for training neural network policies directly.

Why This Matters

Up until now, you've learned value-based methods (DQN, Q-Learning) that learn Q-values and derive policies from them. But what if we want to optimize the policy directly? We need a way to compute how changing our policy parameters θ affects our expected return.

The Core Challenge

Imagine you have a neural network policy π_θ(a|s) that outputs action probabilities. You want to maximize the expected return J(θ). The naive approach fails because:

  • The gradient ∇J(θ) depends on how actions affect future states
  • Future states depend on the policy itself
  • This creates a complex dependency chain

The Elegant Solution

The Policy Gradient Theorem proves that:

∇J(θ) ∝ E[∇log π_θ(a|s) · Q^π(s,a)]

In plain English: the gradient of expected return equals the expected gradient of the log-probability of actions, weighted by how good those actions are (their Q-value).

This is beautiful because:

  • We don't need to differentiate through the environment dynamics
  • We only need to know Q-values and policy probabilities
  • It works even when the environment is stochastic or unknown

Think of it like adjusting a recipe: you make dishes (take actions), taste them (get rewards), and increase the probability of ingredient choices that led to tasty results—without needing to understand the chemistry of cooking.

Key Takeaway: The Policy Gradient Theorem provides a practical formula for updating policy parameters by increasing the probability of actions proportional to their value, enabling direct policy optimization without explicit value function models.