On-Policy vs Off-Policy in Policy Gradients
What You'll Learn
You'll understand why vanilla policy gradient methods must collect fresh data from the current policy and why reusing old experience creates problems.
The Core Distinction
In on-policy learning, you must use data collected from your current policy to update that same policy. In off-policy learning, you can learn from data generated by a different policy (like past versions of your policy or even human demonstrations).
Why Policy Gradients Are Naturally On-Policy
Remember the policy gradient theorem relies on estimating gradients using trajectories sampled from your current policy π_θ. The math explicitly assumes the data distribution matches π_θ:
∇J(θ) = E_τ~π_θ [∇log π_θ(a|s) * A(s,a)]
If your replay buffer contains old trajectories from π_θ_old, that expectation becomes invalid—you're computing gradients using the wrong distribution. This is called distribution mismatch.
The Challenge
Think of it like learning to navigate a city. On-policy learning means you must walk around right now with your current understanding and learn from that fresh experience. Off-policy would let you learn from a friend's GPS history—but their route choices reflect their decision-making, not yours, making it harder to improve your own strategy.
Unlike Q-learning (which updates action values and works off-policy naturally), policy gradients directly optimize the policy's probability distribution. When the data comes from a different distribution, your gradient estimates become biased and can destabilize training.
Practical Implications
Basic policy gradient methods like REINFORCE require:
- Collecting full episodes with the current policy
- Computing gradients from that fresh data
- Discarding the data after one update
- Repeating this cycle
This is sample-inefficient but mathematically sound.
Key Takeaway: Vanilla policy gradient methods require on-policy data because the gradient estimator assumes trajectories come from the current policy distribution; using old data introduces distribution mismatch that violates this assumption.