The Critic: Value Function Approximation
What you'll learn: How neural networks estimate state or state-action values to guide policy improvement in actor-critic methods.
The Core Idea
In pure policy gradient methods like REINFORCE, we estimate returns by running complete episodes—this works, but creates high variance. The critic solves this by learning a value function that predicts expected returns from any state, eliminating the need to wait until episode end.
Think of the critic as an experienced coach watching a game. Instead of waiting until the match finishes to say "that move was worth 10 points," the coach can immediately estimate "from this position, you're likely to score 7 points by the end." This real-time feedback dramatically reduces uncertainty.
How It Works
The critic uses a neural network to approximate either:
- V(s): The value of being in state
s(state-value function) - Q(s, a): The value of taking action
ain states(action-value function)
The network takes a state as input and outputs value estimates. It's trained using the same TD learning principles you've seen in Q-learning and SARSA—minimizing the difference between predicted values and observed returns or bootstrapped targets.
Why This Matters
By maintaining a learned value function, the critic provides:
- Lower variance estimates than Monte Carlo returns
- Immediate feedback without waiting for episode completion
- A baseline for advantage calculation that tells the actor whether an action was better or worse than average
The critic doesn't choose actions—that's still the actor's job. It simply evaluates how good states or actions are, enabling the actor to learn more efficiently.
Key Takeaway: The critic uses neural network value function approximation to provide low-variance estimates of expected returns, serving as both a baseline and a guide for policy improvement in actor-critic architectures.