Course contentsShow
← Machine Learning and Deep Learning
Lesson 2279 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Baseline Subtraction and Variance Reduction

See how subtracting a baseline (value function) from returns reduces gradient variance without introducing bias.

Baseline Subtraction and Variance Reduction

What You'll Learn

You'll learn why subtracting a baseline (like a value function) from returns dramatically reduces the variance of policy gradient estimates without introducing bias.

The Core Idea

In policy gradient methods, we estimate gradients using returns (cumulative rewards). The problem? Returns can vary wildly between episodes, making our gradient estimates noisy and training unstable.

The solution: Subtract a baseline from the return. Instead of using the raw return G_t, we use G_t - b(s_t) where b(s_t) is our baseline (often the value function V(s_t)).

Think of it like grading on a curve. If a student scores 75 points, is that good? It depends on the class average. If the average is 50, that's excellent! If it's 90, not so much. By comparing to a baseline, we get meaningful signals about whether an action performed better or worse than expected.

Why This Works

Variance reduction: The baseline "centers" our returns around zero. Actions that exceed expectations get positive advantages; those below get negative advantages. This reduces the magnitude of our gradient updates while preserving their direction.

Unbiased: Mathematically, the baseline cancels out in expectation because it doesn't depend on the action taken. The gradient estimate remains correct on average, just less noisy.

The Practical Impact

In Actor-Critic, the critic estimates V(s_t) as our baseline. Computing A(s_t, a_t) = G_t - V(s_t) gives us the advantage: how much better this action was compared to the average value of being in that state. This advantage-based gradient is far more stable than using raw returns.

Key Takeaway: Subtracting a baseline from returns reduces gradient variance without introducing bias, making policy gradient training faster and more stable—it's like removing the noise to hear the signal clearly.