Course contentsShow
← Machine Learning and Deep Learning
Lesson 2172 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

The TD(0) Update Rule

Deriving the fundamental TD(0) equation and understanding bootstrapping from estimated values.

The TD(0) Update Rule

What you'll learn: How TD(0) updates value estimates by bootstrapping from the next state's estimated value rather than waiting for actual returns.

The Core Idea

Temporal Difference learning with zero steps lookahead—TD(0)—is the simplest form of TD learning. Unlike Monte Carlo methods that wait until an episode ends to update values, TD(0) updates estimates immediately after each step by using the current estimate of the next state's value.

The fundamental TD(0) update equation is:

V(s) ← V(s) + α[R + γV(s') - V(s)]

Where:

  • V(s) is the current state's value estimate
  • α is the learning rate (step size)
  • R is the immediate reward received
  • γ is the discount factor
  • V(s') is the estimated value of the next state
  • The bracketed term [R + γV(s') - V(s)] is called the TD error

Understanding Bootstrapping

The key innovation is bootstrapping: TD(0) uses its own estimate V(s') to update V(s). Think of it like a student who checks their practice problem answer against the answer key's current best guess rather than waiting for the final, verified solution. The estimate improves iteratively as the "answer key" itself gets better.

The TD target is R + γV(s'), which combines the actual observed reward with an estimated future value. This makes TD(0) a hybrid: it samples real experience like Monte Carlo but bootstraps like Dynamic Programming.

Why It Matters

TD(0) learns online from incomplete episodes, making it practical for continuing tasks and environments where waiting isn't feasible. The bootstrapping allows faster propagation of value information backward through states.

Key Takeaway: TD(0) updates value estimates immediately after each step using the TD error—the difference between the current estimate and a bootstrapped target formed from the observed reward plus the discounted next-state estimate.