Course contentsShow
← Machine Learning and Deep Learning
Lesson 2225 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Double DQN: Addressing Overestimation Bias

How decoupling action selection from evaluation reduces systematic Q-value overestimation in standard DQN.

Double DQN: Addressing Overestimation Bias

What You'll Learn

You'll discover how Double DQN fixes a fundamental flaw in standard DQN that causes it to be unrealistically optimistic about future rewards.

The Overestimation Problem in Standard DQN

In standard DQN, when computing the target for the Q-learning update, we use:

target = reward + gamma * max(Q_target(next_state, a))

This max operator does two jobs at once: it selects which action looks best and evaluates how good that action is—using the same Q-network. This creates a systematic bias toward overestimation.

Why? Imagine you're uncertain about several restaurant ratings. If you pick the restaurant with the highest noisy estimate and then trust that same estimate, you'll consistently overestimate quality because you've cherry-picked the lucky noise.

How Double DQN Decouples Selection from Evaluation

Double DQN separates these two jobs:

  1. Selection: Use the online network to pick which action looks best
  2. Evaluation: Use the target network to estimate that action's true value
best_action = argmax(Q_online(next_state, a))
target = reward + gamma * Q_target(next_state, best_action)

Now one network chooses, but a different network evaluates. Their independent errors don't compound—if the online network picks an overestimated action, the target network (updated less frequently) provides a more sober assessment.

Why This Works

The two networks have different "opinions" because they're at different stages of training. Their errors are less correlated, so you don't systematically pick and confirm the same optimistic mistakes.

Key Takeaway: Double DQN reduces Q-value overestimation by using the online network to select actions and the target network to evaluate them, breaking the feedback loop that causes standard DQN to be overly optimistic.