Course contentsShow
← Machine Learning and Deep Learning
Lesson 2275 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

From Pure Policy Gradients to Actor-Critic

Understand why combining value estimation with policy learning reduces variance and improves sample efficiency.

From Pure Policy Gradients to Actor-Critic

What you'll learn: How combining value estimation with policy learning reduces variance and improves sample efficiency in reinforcement learning.

The Problem with Pure Policy Gradients

REINFORCE and similar pure policy gradient methods suffer from high variance. They estimate gradients using full Monte Carlo returns—essentially waiting until the end of each episode to calculate what actually happened. This creates noisy gradient estimates because two identical actions in the same state might lead to very different outcomes due to randomness.

You've already seen that subtracting a baseline helps reduce variance. But what if we could do even better?

The Actor-Critic Insight

Instead of using Monte Carlo returns (which have high variance) or just a simple baseline, we can use a learned value function to estimate expected future rewards. This creates a hybrid approach:

  • Actor: The policy network that selects actions (what you've been doing)
  • Critic: A value function that estimates how good states or actions are (borrowed from value-based methods like Q-learning)

The critic provides a more stable, lower-variance estimate of returns than waiting for actual outcomes. Think of it like a driving instructor (critic) giving immediate feedback to a student driver (actor), rather than waiting until the end of the trip to say "that was bad."

Why This Works

By replacing high-variance Monte Carlo returns with value function estimates, you get:

  1. Lower variance gradients → more stable learning
  2. Bootstrapping → can learn from incomplete episodes
  3. Better sample efficiency → need fewer environment interactions

The actor still learns the policy, but now the critic guides it with less noisy signals about what's working.

Key Takeaway: Actor-Critic methods combine the best of both worlds—policy optimization from policy gradients and stable value estimation from value-based methods—to achieve lower variance and better sample efficiency than pure policy gradients alone.