Generalized Advantage Estimation (GAE)
You'll learn: How GAE creates a smooth dial between bias and variance in advantage estimates by blending advantages across multiple timesteps.
The Core Problem
In Actor-Critic methods, we've seen how advantage functions help reduce variance. But there's a tension: 1-step TD advantages (A = r + γV(s') - V(s)) have low variance but high bias (we rely heavily on our imperfect value function). Monte Carlo advantages (full returns) have no bias but sky-high variance. We need something in between.
How GAE Works
Generalized Advantage Estimation creates an exponentially-weighted average of n-step advantages. Think of it like blending coffee strengths: instead of choosing pure espresso (1-step) or weak drip (full episode), you create a custom blend.
The GAE formula introduces a parameter λ (lambda, between 0 and 1):
GAE(λ) = (1-λ) * [A₁ + λA₂ + λ²A₃ + λ³A₄ + ...]
Where each Aₙ represents the n-step advantage. When λ=0, you get pure 1-step TD (high bias, low variance). When λ=1, you get Monte Carlo (no bias, high variance). Values like λ=0.95 give you the best of both worlds.
Why This Matters
GAE gives you a tunable bias-variance knob. In practice, λ ≈ 0.95-0.99 works well because:
- Recent rewards get more weight (exponential decay)
- You incorporate information beyond one step
- Variance stays manageable while reducing bias
It's the secret sauce behind algorithms like PPO and modern A3C implementations, making them far more stable than pure REINFORCE.
Key Takeaway: GAE uses exponential weighting with parameter λ to smoothly interpolate between biased-but-stable 1-step advantages and unbiased-but-noisy Monte Carlo returns, giving you precise control over the bias-variance tradeoff.