Advantage Functions in Actor-Critic
You'll learn how advantage functions reduce variance in actor-critic methods while keeping gradient estimates unbiased.
The Core Idea
In actor-critic methods, we update our policy using gradients, but these gradients can be extremely noisy. The advantage function is a clever mathematical trick that tells us how much better (or worse) an action is compared to the average action we'd take in that state.
Instead of using the full Q-value Q(s,a) to guide our policy updates, we use:
A(s,a) = Q(s,a) - V(s)
Where:
- Q(s,a) is the expected return from taking action
ain states - V(s) is the expected return from state
sunder our current policy - A(s,a) is the advantage—how much better action
ais than average
Why This Matters
Think of it like grading on a curve. If everyone in class scores around 70, getting a 75 isn't that impressive—your advantage is only +5. But if you're comparing raw scores, 75 sounds decent. The advantage function removes the "baseline noise" of what's typical in that state.
When we subtract V(s), we're asking: "Was this action better or worse than what we'd normally do here?" This reduces variance because we're measuring relative performance, not absolute returns, which can vary wildly.
Crucially, this subtraction doesn't introduce bias—the expected advantage of our policy's actions is zero, so we're still estimating the true gradient direction.
In Practice
The critic learns both Q(s,a) and V(s), or more commonly, just V(s) and estimates Q(s,a) from observed rewards. The actor then uses these advantage estimates to update the policy more efficiently.
Key Takeaway: The advantage function A(s,a) = Q(s,a) - V(s) dramatically reduces gradient variance in actor-critic methods by measuring relative action quality rather than absolute returns, leading to faster and more stable learning.