Target Network: Stabilizing Q-Learning
What You'll Learn
You'll understand why using a separate, frozen target network prevents training instability and oscillations when learning Q-values with deep neural networks.
The Moving Target Problem
In standard Q-learning, you update your Q-network by minimizing the difference between your current Q-value prediction and a "target" value. The problem? That target depends on the same network you're trying to update:
target = reward + γ * max Q(next_state, action)
Imagine trying to shoot an arrow at a target that moves every time you adjust your aim. As you update your network weights, the target values shift too, creating a feedback loop. Your Q-values can oscillate wildly or even diverge completely—the network chases its own tail.
How Target Networks Solve This
A target network is a frozen copy of your main Q-network. Instead of computing targets using the network you're currently training, you use this separate, stable copy:
target = reward + γ * max Q_target(next_state, action)
The target network stays fixed for many training steps (e.g., 10,000 updates), then gets synchronized with the main network by copying its weights. This creates a stable target that doesn't shift with every gradient update.
The Analogy
Think of it like learning archery with a stationary target versus a moving one. The stationary target (frozen target network) lets you gradually improve your aim. Once you've practiced enough, you move the target to a new position (update the target network) based on your improved skill level.
Key Takeaway: The target network acts as a stable reference point during training, preventing the destructive feedback loop where changing Q-values destabilize their own learning targets, dramatically improving DQN convergence and stability.