Course contentsShow
← Machine Learning and Deep Learning
Lesson 2307 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Value Function Learning in PPO

Training a value network alongside the policy to estimate state values for advantage computation.

Value Function Learning in PPO

What you'll learn: How PPO trains a separate value network alongside the policy to provide better advantage estimates and guide learning.

The Core Idea

In PPO, we don't just learn a policy—we simultaneously train a value network (the "critic") that predicts how good each state is. This value function serves two crucial purposes: it helps us compute advantages for the policy gradient, and it acts as a baseline to reduce variance in our updates.

Think of it like having a coach alongside an athlete. The athlete (policy) takes actions, but the coach (value function) constantly evaluates "how well positioned are we right now?" This evaluation helps the athlete understand whether their recent actions made things better or worse than expected.

How It Works

The value network V(s) takes a state as input and outputs a scalar estimate of the expected return from that state. During training:

  1. Collect experience following the current policy
  2. Compute returns (actual rewards observed)
  3. Update the value network to better predict those returns using mean squared error
  4. Use the value estimates to calculate advantages: A(s,a) = Return - V(s)

The advantage tells us "was this action better or worse than what we typically expect in this state?" This is exactly what the policy gradient needs—not absolute goodness, but relative improvement.

Why Both Networks?

By learning the value function alongside the policy, PPO gets more stable and efficient training. The value function learns from every state visited, providing increasingly accurate baselines that help the policy focus its updates on genuinely surprising outcomes rather than just noisy rewards.

Key Takeaway: PPO trains a value network in parallel with the policy to estimate state values, enabling better advantage computation and more stable policy updates through variance reduction.