Course contentsShow
← Machine Learning and Deep Learning
Lesson 2212 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

DQN Loss Function Derivation

The temporal difference error as a loss function and how it connects to Bellman optimality.

DQN Loss Function Derivation

What you'll learn: How DQN transforms the temporal difference error into a trainable loss function that drives the neural network toward Bellman optimality.

The Core Idea

In Q-Learning, we updated Q-values directly using the temporal difference (TD) error. In DQN, we can't directly update a table—we have a neural network with weights θ. The DQN loss function is designed to minimize the TD error across batches of experiences, effectively teaching the network to satisfy the Bellman optimality equation.

From TD Error to Loss

Recall the Q-Learning update relied on the TD error:

TD error = (reward + γ · max Q(s', a')) - Q(s, a)

In DQN, we turn this into a mean squared error (MSE) loss:

L(θ) = E[(Target - Q(s, a; θ))²]

Where:

  • Target = r + γ · max_a' Q(s', a'; θ⁻) (using target network parameters θ⁻)
  • Q(s, a; θ) is the current network's prediction

Why This Works

Think of it like teaching a student: you show them the "correct answer" (the target based on Bellman optimality) and penalize them proportionally to how far off their guess is. Squaring the error ensures large mistakes are heavily penalized.

The target represents the right-hand side of the Bellman optimality equation, while Q(s, a; θ) is the left-hand side. By minimizing their difference, we're forcing the network to satisfy Q*(s, a) = r + γ · max Q*(s', a').

The target network (θ⁻) provides stable targets—without it, we'd be "chasing a moving target" since both sides of the equation would change simultaneously during training.

Key Takeaway: The DQN loss function is the squared TD error, directly connecting gradient descent optimization to the Bellman optimality principle by minimizing the difference between predicted Q-values and bootstrapped targets.