Course contentsShow
← Machine Learning and Deep Learning
Lesson 2226 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Double DQN Implementation

Implementing the Double DQN update rule by using the online network for action selection and target for evaluation.

Double DQN Implementation

What you'll learn: How to implement the Double DQN algorithm by splitting action selection and evaluation between two networks to reduce overestimation bias.

The Core Idea

In standard DQN, we use the target network to both select the best action and evaluate its value—this causes overestimation because we're picking the action with the maximum noisy Q-value, then using that same noisy estimate. Double DQN fixes this by using the online network (the one being trained) to choose which action looks best, then using the target network to evaluate how good that action actually is.

Think of it like getting a second opinion: your eager friend (online network) suggests what to do, but your cautious friend (target network) estimates whether it's actually a good idea.

The Implementation Change

The standard DQN target is:

y = r + γ * max_a' Q_target(s', a')

In Double DQN, we split this into two steps:

# Step 1: Online network selects the best action
best_actions = online_network(next_states).argmax(dim=1)

# Step 2: Target network evaluates that action
next_q_values = target_network(next_states).gather(1, best_actions.unsqueeze(1))

# Compute target
targets = rewards + gamma * next_q_values * (1 - dones)

The key difference: argmax uses the online network, but the Q-value lookup uses the target network.

Why This Matters

This decoupling prevents the agent from being overly optimistic about actions that just happen to have high noisy estimates. The online network proposes, the target network disposes—reducing systematic overestimation.

Key Takeaway: Double DQN implementation requires only a small change—use the online network for action selection (argmax) and the target network for value estimation—but significantly improves learning stability and final performance.