Course contentsShow
← Machine Learning and Deep Learning
Lesson 2181 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

N-Step TD Methods

Extending TD learning to use n-step returns, interpolating between TD(0) and Monte Carlo.

N-Step TD Methods

What you'll learn: How to bridge the gap between single-step TD learning and full Monte Carlo by using multi-step returns.

The Core Idea

So far you've seen TD(0) (which uses just one step to estimate value) and Monte Carlo (which waits until the end of an episode). N-step TD methods give you the best of both worlds by looking ahead n steps into the future before bootstrapping.

Instead of updating based on just the next reward like TD(0):

V(S_t) ← V(S_t) + α[R_{t+1} + γV(S_{t+1}) - V(S_t)]

N-step TD uses the sum of n actual rewards before bootstrapping:

V(S_t) ← V(S_t) + α[R_{t+1} + γR_{t+2} + ... + γ^{n-1}R_{t+n} + γ^n V(S_{t+n}) - V(S_t)]

The Spectrum of Methods

Think of it as a continuum:

  • n=1: Pure TD(0) — fast updates, high bias
  • n=∞: Pure Monte Carlo — no bias, high variance
  • n=3,5,7...: A middle ground that balances bias and variance

The n-step return captures n steps of real experience, then bootstraps from the estimated value at step n. This reduces variance compared to Monte Carlo (you don't wait forever) while reducing bias compared to TD(0) (you use more real rewards).

Why It Matters

N-step methods let you tune the bias-variance tradeoff. In practice, values like n=3 or n=5 often work better than either extreme. You get faster, more stable learning by choosing the right horizon.

Key Takeaway: N-step TD methods interpolate between TD(0) and Monte Carlo by looking n steps ahead, giving you a tunable balance between bias and variance in value estimates.