Course contentsShow
← Machine Learning and Deep Learning
Lesson 2242 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Computing Target Q-Values

Calculate TD targets using the target network and Bellman equation: target = reward + gamma * max Q(next_state).

Computing Target Q-Values

What you'll learn: How to calculate the TD (temporal difference) targets that guide your Q-network's learning using the target network and Bellman equation.

Why Target Q-Values Matter

In Deep Q-Learning, your agent learns by minimizing the difference between its current Q-value predictions and a target value. But what exactly is that target? It's not a label handed to you like in supervised learning—you have to compute it yourself using the Bellman equation.

The Core Idea

Think of the target Q-value as your "best guess" of what the true Q-value should be. You construct it from two pieces:

  1. The immediate reward you just received from taking an action
  2. The discounted value of the best future action from your next state

The formula is elegantly simple:

target = reward + gamma * max Q_target(next_state)

Where:

  • reward is what you got immediately
  • gamma (γ) is your discount factor (typically 0.99)
  • max Q_target(next_state) is the highest Q-value your target network predicts for the next state

Why Use the Target Network?

You use the target network (not your main Q-network) to compute the max Q-value for stability. This prevents a "chasing your own tail" problem where your network would be learning from its own rapidly changing predictions.

The Practical Flow

When training on a batch of experiences (state, action, reward, next_state, done), you compute targets for each transition, then train your main network to predict these targets. If the episode ended (done=True), the target simplifies to just the reward since there's no future.

Key Takeaway: Target Q-values combine immediate rewards with discounted future value estimates from the target network, giving your Q-network a stable learning signal via the Bellman equation.