Course contentsShow
← Machine Learning and Deep Learning
Lesson 2280 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Temporal Difference Learning in the Critic

Update the critic using TD errors to bootstrap value estimates from incomplete trajectories.

Temporal Difference Learning in the Critic

What you'll learn: How the critic in actor-critic methods uses TD errors to learn value estimates without waiting for complete episode returns.

The Core Idea

In actor-critic architectures, the critic estimates the value function to provide better feedback to the actor. Instead of waiting until an episode ends to compute full returns (like REINFORCE does), the critic uses Temporal Difference (TD) learning to update its estimates immediately after each step.

The TD approach works by bootstrapping: the critic uses its current estimate of the next state's value to update its estimate of the current state's value. This is the same TD learning you've seen in Q-learning and SARSA, but now applied to training the critic network.

How It Works

After taking action a in state s, receiving reward r, and transitioning to state s', the critic computes the TD error:

δ = r + γ·V(s') - V(s)

This TD error measures how wrong the critic's prediction was. The critic then updates its parameters to minimize this error, typically using gradient descent on the squared TD error loss.

Why This Matters

Bootstrapping means the critic doesn't need complete trajectories. It can learn online, step-by-step, making training faster and more sample-efficient than waiting for Monte Carlo returns. The TD error also serves double duty: it trains the critic and provides the advantage estimate that guides the actor's policy updates.

Think of it like a weather forecaster who adjusts their model after each day based on how yesterday's prediction compared to today's forecast, rather than waiting until the end of the year to evaluate accuracy.

Key Takeaway: The critic uses TD errors to bootstrap value estimates from incomplete trajectories, enabling faster, online learning while simultaneously providing advantage signals for policy improvement.