Course contentsShow
← Machine Learning and Deep Learning
Lesson 2258 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Policy Gradient with Value Function Baseline

Learning a state-value function simultaneously to serve as a baseline for policy updates.

Policy Gradient with Value Function Baseline

What You'll Learn

You'll learn how to train a value function in parallel with your policy to create a more effective baseline for reducing gradient variance.

The Core Idea

In the previous lesson, you saw that subtracting a baseline from returns reduces variance without introducing bias. The simplest baseline is a constant (like the average return), but we can do much better: learn a state-value function V(s) that estimates expected returns from each state.

Instead of using a fixed baseline for all states, we use V(s) as a state-dependent baseline. This means:

  • In "good" states (high V(s)), we need larger returns to consider an action advantageous
  • In "difficult" states (low V(s)), even modest returns signal a good action

The advantage becomes: A(s,a) = Q(s,a) - V(s), which we approximate as (G_t - V(s_t)) using the actual return G_t from our episode.

Why This Works Better

Think of it like grading students: instead of comparing everyone to the class average (fixed baseline), you compare each student to their own past performance (state-dependent baseline). A struggling student improving from 60% to 70% might deserve more credit than a top student dropping from 95% to 90%.

The Training Process

You now train two networks simultaneously:

  1. Policy network π(a|s;θ): Updated using policy gradients with the advantage
  2. Value network V(s;w): Updated using standard value function learning (like TD or Monte Carlo)

The value network learns to predict returns, then serves as the baseline for policy updates. Both networks improve together throughout training.

Key Takeaway: Learning a value function as a baseline dramatically reduces policy gradient variance by providing state-specific expectations, making each update more reliable and accelerating learning compared to fixed or zero baselines.