Target Network Updates
What you'll learn: How to synchronize your target network with the Q-network using hard or soft update strategies to maintain training stability.
The Core Idea
You've already learned that DQN uses a separate target network to compute stable Q-value targets during training. But there's a critical question: when and how do you update this target network to reflect what the main Q-network has learned?
There are two main strategies:
Hard Updates (Periodic Copy)
Every C training steps (typically 1,000–10,000), you completely copy all weights from the Q-network to the target network:
θ_target ← θ
Think of this like updating a textbook edition: every semester, you replace the old edition with the new one entirely. The target network stays frozen between updates, providing consistent targets, then suddenly "catches up" all at once.
Soft Updates (Polyak Averaging)
After every training step (or every few steps), you slowly blend the target network toward the Q-network using a small factor τ (tau), typically 0.001–0.01:
θ_target ← τ · θ + (1 - τ) · θ_target
This is like gradually revising your textbook: each update incorporates a tiny bit of new knowledge. With τ = 0.001, the target network shifts only 0.1% toward the current Q-network each update, creating extremely smooth tracking.
Why It Matters
Hard updates are simpler and commonly used in Atari DQN implementations. Soft updates (popularized by DDPG and other actor-critic methods) often provide smoother, more stable learning but require tuning τ. Both approaches prevent the "moving target" problem while ensuring your target network eventually learns from improvements.
Key Takeaway: Target network updates synchronize learning progress—use hard updates (periodic full copy) for simplicity or soft updates (Polyak averaging) for smoother, incremental tracking of the Q-network's improvements.