Course contentsShow
← Machine Learning and Deep Learning
Lesson 2269 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Baseline Subtraction for Variance Reduction

Using state-value baselines to reduce gradient variance without introducing bias in REINFORCE.

Baseline Subtraction for Variance Reduction

What You'll Learn

You'll learn how subtracting a baseline from returns in REINFORCE reduces gradient variance dramatically without biasing the gradient estimates.

The Core Problem

REINFORCE works, but its gradients are noisy. When we compute the policy gradient using returns (G_t), some episodes might have high returns and others low returns—even when following the same policy. This variance slows learning and makes training unstable.

How Baseline Subtraction Works

The key insight: we can subtract any function b(s) from our returns as long as it doesn't depend on the action taken. The gradient becomes:

∇J(θ) = E[∇log π(a|s, θ) · (G_t - b(s))]

Why this works: The baseline term doesn't introduce bias because:

E[∇log π(a|s, θ) · b(s)] = 0

The gradient of the log probability, when averaged over all actions, sums to zero. So mathematically, the baseline "disappears" in expectation—but it dramatically reduces variance in practice.

The Best Baseline: State Value

The most effective baseline is the state-value function V(s). By subtracting V(s) from returns, we're essentially asking: "Was this return better or worse than expected?" This centers our gradient updates around zero, reducing wild swings.

Instead of: "This action got return +100, so increase its probability"
We get: "This action got +100, but we expected +95, so increase its probability a little"

Intuitive Analogy

Imagine rating a restaurant. Without a baseline, you might give 4 stars and think "pretty good!" But if you expected 5 stars based on reviews (your baseline), that same 4-star experience feels disappointing. The baseline gives you context, making your judgment more precise and less noisy.

Key Takeaway: Subtracting a state-value baseline from returns in REINFORCE keeps gradient estimates unbiased while dramatically reducing variance, leading to faster and more stable policy learning.