Loss Function and Backpropagation
What You'll Learn
You'll learn how to compute the Mean Squared Error (MSE) loss between your Q-network's predictions and target values, then use backpropagation to update the network weights.
The Core Idea
Think of the loss function as a report card measuring how wrong your Q-network's predictions are. In DQN, you've already computed target Q-values from the Bellman equation. Now you need to compare them against what your network currently predicts and fix the mistakes.
MSE Loss calculates the average squared difference between predicted Q-values and targets:
Loss = (1/batch_size) * Σ(Q_predicted - Q_target)²
This loss is always positive and grows larger when predictions are further from targets.
Implementing the Loss
In PyTorch, computing and optimizing the loss looks like this:
import torch.nn.functional as F
# q_values: [batch_size, num_actions] from your Q-network
# actions: [batch_size] - which actions were actually taken
# target_q_values: [batch_size] - computed from target network
# Select Q-values for actions that were taken
q_values_for_actions = q_values.gather(1, actions.unsqueeze(1)).squeeze(1)
# Compute MSE loss
loss = F.mse_loss(q_values_for_actions, target_q_values)
# Backpropagation
optimizer.zero_grad() # Clear old gradients
loss.backward() # Compute new gradients
optimizer.step() # Update weights
Backpropagation automatically calculates how much each network parameter contributed to the error and adjusts them via gradient descent to reduce future errors.
Why This Matters
This three-step process (compute loss → backpropagate → update weights) is the heartbeat of DQN training. Each iteration slightly improves your Q-network's ability to estimate action values accurately.
Key Takeaway: The MSE loss quantifies prediction errors, and backpropagation uses these errors to update Q-network weights through gradient descent, gradually improving Q-value estimates.