The Policy Gradient Theorem
What you'll learn: How to mathematically derive gradients of expected return with respect to policy parameters—the foundation for training neural network policies directly.
Why This Matters
Up until now, you've learned value-based methods (DQN, Q-Learning) that learn Q-values and derive policies from them. But what if we want to optimize the policy directly? We need a way to compute how changing our policy parameters θ affects our expected return.
The Core Challenge
Imagine you have a neural network policy π_θ(a|s) that outputs action probabilities. You want to maximize the expected return J(θ). The naive approach fails because:
- The gradient ∇J(θ) depends on how actions affect future states
- Future states depend on the policy itself
- This creates a complex dependency chain
The Elegant Solution
The Policy Gradient Theorem proves that:
∇J(θ) ∝ E[∇log π_θ(a|s) · Q^π(s,a)]
In plain English: the gradient of expected return equals the expected gradient of the log-probability of actions, weighted by how good those actions are (their Q-value).
This is beautiful because:
- We don't need to differentiate through the environment dynamics
- We only need to know Q-values and policy probabilities
- It works even when the environment is stochastic or unknown
Think of it like adjusting a recipe: you make dishes (take actions), taste them (get rewards), and increase the probability of ingredient choices that led to tasty results—without needing to understand the chemistry of cooking.
Key Takeaway: The Policy Gradient Theorem provides a practical formula for updating policy parameters by increasing the probability of actions proportional to their value, enabling direct policy optimization without explicit value function models.