Advantage Function in Policy Gradients
What You'll Learn
You'll learn how the advantage function measures how much better an action is compared to average, producing more informative gradients for policy learning.
The Problem with Raw Returns
In policy gradients, we update parameters based on actions and their returns. But a return of +10 doesn't tell us much—was that action genuinely good, or is +10 just typical for this state? Without context, our gradient updates lack direction.
The Advantage Function: Relative Quality
The advantage function provides that context:
A(s, a) = Q(s, a) - V(s)
Where:
Q(s, a)is the action-value (expected return from taking actionain states)V(s)is the state-value (expected return from statesunder the current policy)A(s, a)tells us how much better or worse actionais compared to the average action
Why This Matters
Think of it like grading on a curve. If everyone in class scores around 80, getting an 85 means you're above average (positive advantage). But if the average is 95, that same 85 is below par (negative advantage).
In policy gradients, using advantages instead of raw returns means:
- Positive advantage → strengthen this action's probability
- Negative advantage → weaken this action's probability
- Near-zero advantage → minimal update (action is typical)
This dramatically reduces variance because we're measuring relative quality, not absolute returns that vary wildly between episodes.
Practical Benefit
Instead of gradient = log_prob(a|s) × return, we use:
gradient = log_prob(a|s) × A(s, a)
This centers our updates around what's better-than-average, making learning more stable and efficient.
Key Takeaway: The advantage function A(s,a) = Q(s,a) - V(s) measures how much better an action is than the state's average, providing centered, lower-variance gradients that guide policy improvement more effectively.