Baseline Subtraction and Variance Reduction
What You'll Learn
You'll learn why subtracting a baseline (like a value function) from returns dramatically reduces the variance of policy gradient estimates without introducing bias.
The Core Idea
In policy gradient methods, we estimate gradients using returns (cumulative rewards). The problem? Returns can vary wildly between episodes, making our gradient estimates noisy and training unstable.
The solution: Subtract a baseline from the return. Instead of using the raw return G_t, we use G_t - b(s_t) where b(s_t) is our baseline (often the value function V(s_t)).
Think of it like grading on a curve. If a student scores 75 points, is that good? It depends on the class average. If the average is 50, that's excellent! If it's 90, not so much. By comparing to a baseline, we get meaningful signals about whether an action performed better or worse than expected.
Why This Works
Variance reduction: The baseline "centers" our returns around zero. Actions that exceed expectations get positive advantages; those below get negative advantages. This reduces the magnitude of our gradient updates while preserving their direction.
Unbiased: Mathematically, the baseline cancels out in expectation because it doesn't depend on the action taken. The gradient estimate remains correct on average, just less noisy.
The Practical Impact
In Actor-Critic, the critic estimates V(s_t) as our baseline. Computing A(s_t, a_t) = G_t - V(s_t) gives us the advantage: how much better this action was compared to the average value of being in that state. This advantage-based gradient is far more stable than using raw returns.
Key Takeaway: Subtracting a baseline from returns reduces gradient variance without introducing bias, making policy gradient training faster and more stable—it's like removing the noise to hear the signal clearly.