DQN Loss Function Derivation
What you'll learn: How DQN transforms the temporal difference error into a trainable loss function that drives the neural network toward Bellman optimality.
The Core Idea
In Q-Learning, we updated Q-values directly using the temporal difference (TD) error. In DQN, we can't directly update a table—we have a neural network with weights θ. The DQN loss function is designed to minimize the TD error across batches of experiences, effectively teaching the network to satisfy the Bellman optimality equation.
From TD Error to Loss
Recall the Q-Learning update relied on the TD error:
TD error = (reward + γ · max Q(s', a')) - Q(s, a)
In DQN, we turn this into a mean squared error (MSE) loss:
L(θ) = E[(Target - Q(s, a; θ))²]
Where:
- Target = r + γ · max_a' Q(s', a'; θ⁻) (using target network parameters θ⁻)
- Q(s, a; θ) is the current network's prediction
Why This Works
Think of it like teaching a student: you show them the "correct answer" (the target based on Bellman optimality) and penalize them proportionally to how far off their guess is. Squaring the error ensures large mistakes are heavily penalized.
The target represents the right-hand side of the Bellman optimality equation, while Q(s, a; θ) is the left-hand side. By minimizing their difference, we're forcing the network to satisfy Q*(s, a) = r + γ · max Q*(s', a').
The target network (θ⁻) provides stable targets—without it, we'd be "chasing a moving target" since both sides of the equation would change simultaneously during training.
Key Takeaway: The DQN loss function is the squared TD error, directly connecting gradient descent optimization to the Bellman optimality principle by minimizing the difference between predicted Q-values and bootstrapped targets.