Course contentsShow
← Machine Learning and Deep Learning
Lesson 2256 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Baselines for Variance Reduction

Subtracting baseline functions from returns to reduce gradient variance without introducing bias.

Baselines for Variance Reduction

What you'll learn: How subtracting a baseline function from returns dramatically reduces gradient variance in policy gradient methods without biasing your estimates.

The Problem with Raw Returns

In policy gradient methods, we estimate gradients using sampled trajectories. The formula involves multiplying the gradient of log-probabilities by returns (rewards-to-go). However, these returns can vary wildly between episodes—even for the same actions—creating massive variance in our gradient estimates. High variance means unstable, slow learning.

The Baseline Solution

A baseline is any function b(s) that depends only on the state (not the action). We subtract it from our returns:

∇J(θ) ≈ (1/N) Σ ∇log π(a|s) · [G_t - b(s)]

Instead of using raw return G_t, we use G_t - b(s). This reduces variance by "centering" the returns around zero.

Why It Works Without Bias

The mathematical beauty: subtracting a state-dependent baseline doesn't introduce bias because:

E[∇log π(a|s) · b(s)] = Σ π(a|s) · ∇log π(a|s) · b(s) = 0

The baseline gets multiplied by something that sums to zero across all actions, so it disappears in expectation.

Intuitive Analogy

Think of grading students. Raw scores (60, 65, 70) have high variance. But if you subtract the class average (65), you get centered scores (-5, 0, +5) that show relative performance more clearly. You haven't changed who's above or below average—just made the signal clearer.

The most common baseline? The value function V(s), which estimates expected returns from each state. This transforms returns into advantages: how much better an action was than average.

Key Takeaway: Baselines reduce policy gradient variance without introducing bias by subtracting state-dependent functions from returns, with the value function being the most effective choice.