Introduction to Temporal Difference Learning
What you'll learn: How Temporal Difference (TD) learning combines the best aspects of Monte Carlo methods and Dynamic Programming to estimate value functions efficiently in reinforcement learning.
The Bridge Between Two Worlds
You've already learned two powerful approaches for estimating value functions:
- Dynamic Programming (like Policy Iteration and Value Iteration): Requires a complete model of the environment (transition probabilities), uses bootstrapping to update estimates based on other estimates
- Monte Carlo methods: Model-free, learns from complete episodes, but must wait until the end of an episode to update
Temporal Difference learning bridges these approaches — it learns directly from experience like Monte Carlo (no model needed), but updates estimates immediately after each step using bootstrapping like Dynamic Programming.
The Core Idea
TD learning asks: "Why wait until the end of an episode to learn?"
Instead of waiting for the full return, TD methods update value estimates using a combination of:
- The immediate reward you just received
- Your current estimate of future value (bootstrapping)
Think of it like updating your restaurant rating mid-meal rather than waiting to finish. After tasting the appetizer and seeing the main course, you can already adjust your opinion — you don't need to wait for dessert and the bill.
Why This Matters
TD learning is online and incremental: it learns from incomplete sequences, making it practical for continuing tasks and environments where episodes are very long or non-existent. This flexibility makes TD methods the foundation for many modern RL algorithms you'll encounter next.
Key Takeaway: Temporal Difference learning is a model-free method that updates value estimates after every step by combining immediate rewards with bootstrapped future value estimates, offering the practical advantages of both Monte Carlo and Dynamic Programming approaches.