Course contentsShow
← Machine Learning and Deep Learning
Lesson 2231 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Multi-Step Returns: n-Step DQN

Using multi-step bootstrapping to reduce bias and propagate rewards faster through the Q-network.

Multi-Step Returns: n-Step DQN

What You'll Learn

You'll learn how multi-step bootstrapping helps DQN agents learn faster by propagating reward information more efficiently through the Q-network.

The Core Problem with 1-Step Returns

Standard DQN uses 1-step TD learning: it updates Q-values based only on the immediate reward plus the estimated value of the next state. While this reduces variance compared to Monte Carlo methods, it can be slow to propagate rewards backward through long action sequences.

Think of it like learning you got a promotion: with 1-step learning, you only update your belief about yesterday's actions. But what about all the good work you did last week or last month that led to this outcome?

Multi-Step Bootstrapping

n-Step DQN looks ahead n steps before bootstrapping from the Q-network estimate:

Instead of: r₀ + γ * max Q(s₁, a)

Use: r₀ + γr₁ + γ²r₂ + ... + γⁿ⁻¹rₙ₋₁ + γⁿ * max Q(sₙ, a)

This accumulates n actual rewards before relying on the value estimate.

Benefits

  • Faster credit assignment: Rewards propagate backward through n states in a single update
  • Reduced bias: Less reliance on potentially inaccurate Q-value bootstrapping
  • Better sample efficiency: Each experience teaches the agent about multiple state-action transitions

The Tradeoff

Higher n means lower bias but higher variance (you're summing more stochastic rewards). Typical values: n=3 to n=5 work well in practice.

You can store n-step returns in your replay buffer by keeping sequences of transitions and computing the n-step target when sampling.

Key Takeaway: n-Step DQN uses multi-step bootstrapping to accumulate several real rewards before estimating future value, reducing bias and accelerating learning by propagating reward signals faster through the network.