Course contentsShow
← Machine Learning and Deep Learning
Lesson 2243 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Loss Function and Backpropagation

Implement the MSE loss between predicted Q-values and targets, and perform gradient descent to update the Q-network.

Loss Function and Backpropagation

What You'll Learn

You'll learn how to compute the Mean Squared Error (MSE) loss between your Q-network's predictions and target values, then use backpropagation to update the network weights.

The Core Idea

Think of the loss function as a report card measuring how wrong your Q-network's predictions are. In DQN, you've already computed target Q-values from the Bellman equation. Now you need to compare them against what your network currently predicts and fix the mistakes.

MSE Loss calculates the average squared difference between predicted Q-values and targets:

Loss = (1/batch_size) * Σ(Q_predicted - Q_target)²

This loss is always positive and grows larger when predictions are further from targets.

Implementing the Loss

In PyTorch, computing and optimizing the loss looks like this:

import torch.nn.functional as F

# q_values: [batch_size, num_actions] from your Q-network
# actions: [batch_size] - which actions were actually taken
# target_q_values: [batch_size] - computed from target network

# Select Q-values for actions that were taken
q_values_for_actions = q_values.gather(1, actions.unsqueeze(1)).squeeze(1)

# Compute MSE loss
loss = F.mse_loss(q_values_for_actions, target_q_values)

# Backpropagation
optimizer.zero_grad()  # Clear old gradients
loss.backward()        # Compute new gradients
optimizer.step()       # Update weights

Backpropagation automatically calculates how much each network parameter contributed to the error and adjusts them via gradient descent to reduce future errors.

Why This Matters

This three-step process (compute loss → backpropagate → update weights) is the heartbeat of DQN training. Each iteration slightly improves your Q-network's ability to estimate action values accurately.

Key Takeaway: The MSE loss quantifies prediction errors, and backpropagation uses these errors to update Q-network weights through gradient descent, gradually improving Q-value estimates.