Policy Gradient with Value Function Baseline
What You'll Learn
You'll learn how to train a value function in parallel with your policy to create a more effective baseline for reducing gradient variance.
The Core Idea
In the previous lesson, you saw that subtracting a baseline from returns reduces variance without introducing bias. The simplest baseline is a constant (like the average return), but we can do much better: learn a state-value function V(s) that estimates expected returns from each state.
Instead of using a fixed baseline for all states, we use V(s) as a state-dependent baseline. This means:
- In "good" states (high V(s)), we need larger returns to consider an action advantageous
- In "difficult" states (low V(s)), even modest returns signal a good action
The advantage becomes: A(s,a) = Q(s,a) - V(s), which we approximate as (G_t - V(s_t)) using the actual return G_t from our episode.
Why This Works Better
Think of it like grading students: instead of comparing everyone to the class average (fixed baseline), you compare each student to their own past performance (state-dependent baseline). A struggling student improving from 60% to 70% might deserve more credit than a top student dropping from 95% to 90%.
The Training Process
You now train two networks simultaneously:
- Policy network π(a|s;θ): Updated using policy gradients with the advantage
- Value network V(s;w): Updated using standard value function learning (like TD or Monte Carlo)
The value network learns to predict returns, then serves as the baseline for policy updates. Both networks improve together throughout training.
Key Takeaway: Learning a value function as a baseline dramatically reduces policy gradient variance by providing state-specific expectations, making each update more reliable and accelerating learning compared to fixed or zero baselines.