Baselines for Variance Reduction
What you'll learn: How subtracting a baseline function from returns dramatically reduces gradient variance in policy gradient methods without biasing your estimates.
The Problem with Raw Returns
In policy gradient methods, we estimate gradients using sampled trajectories. The formula involves multiplying the gradient of log-probabilities by returns (rewards-to-go). However, these returns can vary wildly between episodes—even for the same actions—creating massive variance in our gradient estimates. High variance means unstable, slow learning.
The Baseline Solution
A baseline is any function b(s) that depends only on the state (not the action). We subtract it from our returns:
∇J(θ) ≈ (1/N) Σ ∇log π(a|s) · [G_t - b(s)]
Instead of using raw return G_t, we use G_t - b(s). This reduces variance by "centering" the returns around zero.
Why It Works Without Bias
The mathematical beauty: subtracting a state-dependent baseline doesn't introduce bias because:
E[∇log π(a|s) · b(s)] = Σ π(a|s) · ∇log π(a|s) · b(s) = 0
The baseline gets multiplied by something that sums to zero across all actions, so it disappears in expectation.
Intuitive Analogy
Think of grading students. Raw scores (60, 65, 70) have high variance. But if you subtract the class average (65), you get centered scores (-5, 0, +5) that show relative performance more clearly. You haven't changed who's above or below average—just made the signal clearer.
The most common baseline? The value function V(s), which estimates expected returns from each state. This transforms returns into advantages: how much better an action was than average.
Key Takeaway: Baselines reduce policy gradient variance without introducing bias by subtracting state-dependent functions from returns, with the value function being the most effective choice.