Course contentsShow
← Machine Learning and Deep Learning
Lesson 2223 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Target Network: Stabilizing Q-Learning

Why a separate frozen target network prevents moving target problems and oscillations during training.

Target Network: Stabilizing Q-Learning

What You'll Learn

You'll understand why using a separate, frozen target network prevents training instability and oscillations when learning Q-values with deep neural networks.

The Moving Target Problem

In standard Q-learning, you update your Q-network by minimizing the difference between your current Q-value prediction and a "target" value. The problem? That target depends on the same network you're trying to update:

target = reward + γ * max Q(next_state, action)

Imagine trying to shoot an arrow at a target that moves every time you adjust your aim. As you update your network weights, the target values shift too, creating a feedback loop. Your Q-values can oscillate wildly or even diverge completely—the network chases its own tail.

How Target Networks Solve This

A target network is a frozen copy of your main Q-network. Instead of computing targets using the network you're currently training, you use this separate, stable copy:

target = reward + γ * max Q_target(next_state, action)

The target network stays fixed for many training steps (e.g., 10,000 updates), then gets synchronized with the main network by copying its weights. This creates a stable target that doesn't shift with every gradient update.

The Analogy

Think of it like learning archery with a stationary target versus a moving one. The stationary target (frozen target network) lets you gradually improve your aim. Once you've practiced enough, you move the target to a new position (update the target network) based on your improved skill level.

Key Takeaway: The target network acts as a stable reference point during training, preventing the destructive feedback loop where changing Q-values destabilize their own learning targets, dramatically improving DQN convergence and stability.