The TD(0) Update Rule
What you'll learn: How TD(0) updates value estimates by bootstrapping from the next state's estimated value rather than waiting for actual returns.
The Core Idea
Temporal Difference learning with zero steps lookahead—TD(0)—is the simplest form of TD learning. Unlike Monte Carlo methods that wait until an episode ends to update values, TD(0) updates estimates immediately after each step by using the current estimate of the next state's value.
The fundamental TD(0) update equation is:
V(s) ← V(s) + α[R + γV(s') - V(s)]
Where:
V(s)is the current state's value estimateαis the learning rate (step size)Ris the immediate reward receivedγis the discount factorV(s')is the estimated value of the next state- The bracketed term
[R + γV(s') - V(s)]is called the TD error
Understanding Bootstrapping
The key innovation is bootstrapping: TD(0) uses its own estimate V(s') to update V(s). Think of it like a student who checks their practice problem answer against the answer key's current best guess rather than waiting for the final, verified solution. The estimate improves iteratively as the "answer key" itself gets better.
The TD target is R + γV(s'), which combines the actual observed reward with an estimated future value. This makes TD(0) a hybrid: it samples real experience like Monte Carlo but bootstraps like Dynamic Programming.
Why It Matters
TD(0) learns online from incomplete episodes, making it practical for continuing tasks and environments where waiting isn't feasible. The bootstrapping allows faster propagation of value information backward through states.
Key Takeaway: TD(0) updates value estimates immediately after each step using the TD error—the difference between the current estimate and a bootstrapped target formed from the observed reward plus the discounted next-state estimate.