Advantage Estimation in PPO
What you'll learn: How PPO uses Generalized Advantage Estimation (GAE) to compute advantages that balance the tradeoff between bias and variance in policy gradient estimates.
Why Advantage Estimation Matters
In PPO, the clipped surrogate objective needs advantage values to determine which actions should be reinforced. Simply put, the advantage tells us "how much better was this action compared to what we typically expect in this state?"
However, computing advantages isn't straightforward—we face a fundamental tradeoff:
- Low bias, high variance: Use full Monte Carlo returns (wait until episode ends)
- High bias, low variance: Use one-step TD estimates (bootstrap immediately)
Generalized Advantage Estimation (GAE)
GAE provides a middle ground by introducing a parameter λ (lambda) that lets you smoothly interpolate between these extremes. Think of λ as a "memory dial"—turning it toward 1 means trusting longer rollouts, while turning it toward 0 means trusting the value function more.
The GAE advantage at time t is computed as:
A^GAE(t) = δ_t + (γλ)δ_{t+1} + (γλ)²δ_{t+2} + ...
where δ_t = r_t + γV(s_{t+1}) - V(s_t) is the TD error.
The Bias-Variance Knob
- λ = 0: Pure one-step TD (high bias, low variance)—relies entirely on your critic's value estimates
- λ = 1: Full Monte Carlo returns (low bias, high variance)—uses actual observed rewards
- λ ≈ 0.95-0.99: Typical sweet spot in practice—balances both concerns
This exponentially-weighted average of TD errors creates smoother, more reliable advantage estimates than either extreme alone.
Key Takeaway: PPO typically uses GAE with λ between 0.95-0.99 to compute advantages that balance bias and variance, producing stable and efficient policy updates by combining immediate TD feedback with longer-horizon return information.