Course contentsShow
← Machine Learning and Deep Learning
Lesson 2257 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Advantage Function in Policy Gradients

Using the advantage A(s,a) = Q(s,a) - V(s) to measure relative action quality for better gradients.

Advantage Function in Policy Gradients

What You'll Learn

You'll learn how the advantage function measures how much better an action is compared to average, producing more informative gradients for policy learning.

The Problem with Raw Returns

In policy gradients, we update parameters based on actions and their returns. But a return of +10 doesn't tell us much—was that action genuinely good, or is +10 just typical for this state? Without context, our gradient updates lack direction.

The Advantage Function: Relative Quality

The advantage function provides that context:

A(s, a) = Q(s, a) - V(s)

Where:

  • Q(s, a) is the action-value (expected return from taking action a in state s)
  • V(s) is the state-value (expected return from state s under the current policy)
  • A(s, a) tells us how much better or worse action a is compared to the average action

Why This Matters

Think of it like grading on a curve. If everyone in class scores around 80, getting an 85 means you're above average (positive advantage). But if the average is 95, that same 85 is below par (negative advantage).

In policy gradients, using advantages instead of raw returns means:

  • Positive advantage → strengthen this action's probability
  • Negative advantage → weaken this action's probability
  • Near-zero advantage → minimal update (action is typical)

This dramatically reduces variance because we're measuring relative quality, not absolute returns that vary wildly between episodes.

Practical Benefit

Instead of gradient = log_prob(a|s) × return, we use:

gradient = log_prob(a|s) × A(s, a)

This centers our updates around what's better-than-average, making learning more stable and efficient.

Key Takeaway: The advantage function A(s,a) = Q(s,a) - V(s) measures how much better an action is than the state's average, providing centered, lower-variance gradients that guide policy improvement more effectively.