N-Step TD Methods
What you'll learn: How to bridge the gap between single-step TD learning and full Monte Carlo by using multi-step returns.
The Core Idea
So far you've seen TD(0) (which uses just one step to estimate value) and Monte Carlo (which waits until the end of an episode). N-step TD methods give you the best of both worlds by looking ahead n steps into the future before bootstrapping.
Instead of updating based on just the next reward like TD(0):
V(S_t) ← V(S_t) + α[R_{t+1} + γV(S_{t+1}) - V(S_t)]
N-step TD uses the sum of n actual rewards before bootstrapping:
V(S_t) ← V(S_t) + α[R_{t+1} + γR_{t+2} + ... + γ^{n-1}R_{t+n} + γ^n V(S_{t+n}) - V(S_t)]
The Spectrum of Methods
Think of it as a continuum:
- n=1: Pure TD(0) — fast updates, high bias
- n=∞: Pure Monte Carlo — no bias, high variance
- n=3,5,7...: A middle ground that balances bias and variance
The n-step return captures n steps of real experience, then bootstraps from the estimated value at step n. This reduces variance compared to Monte Carlo (you don't wait forever) while reducing bias compared to TD(0) (you use more real rewards).
Why It Matters
N-step methods let you tune the bias-variance tradeoff. In practice, values like n=3 or n=5 often work better than either extreme. You get faster, more stable learning by choosing the right horizon.
Key Takeaway: N-step TD methods interpolate between TD(0) and Monte Carlo by looking n steps ahead, giving you a tunable balance between bias and variance in value estimates.