Double DQN: Addressing Overestimation Bias
What You'll Learn
You'll discover how Double DQN fixes a fundamental flaw in standard DQN that causes it to be unrealistically optimistic about future rewards.
The Overestimation Problem in Standard DQN
In standard DQN, when computing the target for the Q-learning update, we use:
target = reward + gamma * max(Q_target(next_state, a))
This max operator does two jobs at once: it selects which action looks best and evaluates how good that action is—using the same Q-network. This creates a systematic bias toward overestimation.
Why? Imagine you're uncertain about several restaurant ratings. If you pick the restaurant with the highest noisy estimate and then trust that same estimate, you'll consistently overestimate quality because you've cherry-picked the lucky noise.
How Double DQN Decouples Selection from Evaluation
Double DQN separates these two jobs:
- Selection: Use the online network to pick which action looks best
- Evaluation: Use the target network to estimate that action's true value
best_action = argmax(Q_online(next_state, a))
target = reward + gamma * Q_target(next_state, best_action)
Now one network chooses, but a different network evaluates. Their independent errors don't compound—if the online network picks an overestimated action, the target network (updated less frequently) provides a more sober assessment.
Why This Works
The two networks have different "opinions" because they're at different stages of training. Their errors are less correlated, so you don't systematically pick and confirm the same optimistic mistakes.
Key Takeaway: Double DQN reduces Q-value overestimation by using the online network to select actions and the target network to evaluate them, breaking the feedback loop that causes standard DQN to be overly optimistic.