Baseline Subtraction for Variance Reduction
What You'll Learn
You'll learn how subtracting a baseline from returns in REINFORCE reduces gradient variance dramatically without biasing the gradient estimates.
The Core Problem
REINFORCE works, but its gradients are noisy. When we compute the policy gradient using returns (G_t), some episodes might have high returns and others low returns—even when following the same policy. This variance slows learning and makes training unstable.
How Baseline Subtraction Works
The key insight: we can subtract any function b(s) from our returns as long as it doesn't depend on the action taken. The gradient becomes:
∇J(θ) = E[∇log π(a|s, θ) · (G_t - b(s))]
Why this works: The baseline term doesn't introduce bias because:
E[∇log π(a|s, θ) · b(s)] = 0
The gradient of the log probability, when averaged over all actions, sums to zero. So mathematically, the baseline "disappears" in expectation—but it dramatically reduces variance in practice.
The Best Baseline: State Value
The most effective baseline is the state-value function V(s). By subtracting V(s) from returns, we're essentially asking: "Was this return better or worse than expected?" This centers our gradient updates around zero, reducing wild swings.
Instead of: "This action got return +100, so increase its probability"
We get: "This action got +100, but we expected +95, so increase its probability a little"
Intuitive Analogy
Imagine rating a restaurant. Without a baseline, you might give 4 stars and think "pretty good!" But if you expected 5 stars based on reviews (your baseline), that same 4-star experience feels disappointing. The baseline gives you context, making your judgment more precise and less noisy.
Key Takeaway: Subtracting a state-value baseline from returns in REINFORCE keeps gradient estimates unbiased while dramatically reducing variance, leading to faster and more stable policy learning.