Course contentsShow
← Machine Learning and Deep Learning
Lesson 2261 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

On-Policy vs Off-Policy in Policy Gradients

Why basic policy gradient methods require on-policy data and the challenges of off-policy learning.

On-Policy vs Off-Policy in Policy Gradients

What You'll Learn

You'll understand why vanilla policy gradient methods must collect fresh data from the current policy and why reusing old experience creates problems.

The Core Distinction

In on-policy learning, you must use data collected from your current policy to update that same policy. In off-policy learning, you can learn from data generated by a different policy (like past versions of your policy or even human demonstrations).

Why Policy Gradients Are Naturally On-Policy

Remember the policy gradient theorem relies on estimating gradients using trajectories sampled from your current policy π_θ. The math explicitly assumes the data distribution matches π_θ:

∇J(θ) = E_τ~π_θ [∇log π_θ(a|s) * A(s,a)]

If your replay buffer contains old trajectories from π_θ_old, that expectation becomes invalid—you're computing gradients using the wrong distribution. This is called distribution mismatch.

The Challenge

Think of it like learning to navigate a city. On-policy learning means you must walk around right now with your current understanding and learn from that fresh experience. Off-policy would let you learn from a friend's GPS history—but their route choices reflect their decision-making, not yours, making it harder to improve your own strategy.

Unlike Q-learning (which updates action values and works off-policy naturally), policy gradients directly optimize the policy's probability distribution. When the data comes from a different distribution, your gradient estimates become biased and can destabilize training.

Practical Implications

Basic policy gradient methods like REINFORCE require:

  • Collecting full episodes with the current policy
  • Computing gradients from that fresh data
  • Discarding the data after one update
  • Repeating this cycle

This is sample-inefficient but mathematically sound.

Key Takeaway: Vanilla policy gradient methods require on-policy data because the gradient estimator assumes trajectories come from the current policy distribution; using old data introduces distribution mismatch that violates this assumption.