TD vs Monte Carlo: Bias-Variance Tradeoff
What you'll learn: You'll understand why TD methods and Monte Carlo methods make different trade-offs between bias and variance when learning value functions.
The Core Trade-off
Both Temporal Difference (TD) and Monte Carlo (MC) methods estimate value functions, but they make fundamentally different choices about when and how to update their estimates.
Monte Carlo methods wait until the end of an episode, observe the actual return, and update values based on complete experience. They are unbiased (the expected update target equals the true value) but suffer from high variance because full returns can vary wildly depending on all the random actions and state transitions that occur.
TD methods update immediately after each step using a bootstrapped estimate—they guess the remaining return using their current value function. This introduces bias (early estimates may be wrong), but dramatically reduces variance because they only depend on one step of randomness, not an entire trajectory.
The Intuition
Think of learning how long your commute takes:
- MC approach: Drive the full route every time, record total time, then update your estimate. Accurate target, but lots of variability from traffic, lights, etc.
- TD approach: After driving one block, estimate remaining time based on your current belief. Less noisy (one block is more predictable), but initially wrong if your beliefs are off.
Convergence Implications
- MC converges to the true values but requires many episodes and can be slow
- TD often converges faster in practice despite bias, because lower variance means more stable learning
- TD can learn online (continuing tasks), while MC requires episode completion
Key Takeaway: TD methods trade bias for reduced variance, often learning faster than Monte Carlo despite bootstrapping from potentially incorrect estimates—a classic bias-variance tradeoff in action.