Target Network Update Strategies
What you'll learn: How to synchronize your target network with the online network using hard updates versus soft (Polyak) updates in Deep Q-Networks.
The Core Problem
You already know that DQN uses a separate target network to stabilize training by providing consistent Q-value targets. But how often and by how much should you update this target network? Update it too frequently and you lose stability; update it too rarely and your targets become stale.
Hard Updates (Original DQN Approach)
The simplest strategy: copy all weights from the online network to the target network every C training steps (e.g., every 10,000 steps).
if step % target_update_frequency == 0:
target_network.load_state_dict(online_network.state_dict())
Analogy: Like replacing your GPS map completely every few hours. You get all new information at once, but nothing changes between updates.
Pros: Simple, proven to work
Cons: Creates discontinuities in target values; targets suddenly "jump"
Soft Updates (Polyak Averaging)
Introduced by DDPG and other algorithms, soft updates blend the networks gradually at every step using parameter τ (tau), typically 0.001–0.01:
# After every training step
for target_param, online_param in zip(target_network.parameters(),
online_network.parameters()):
target_param.data.copy_(τ * online_param.data + (1 - τ) * target_param.data)
Analogy: Like your GPS gradually updating the map as you drive, smoothly incorporating new road data rather than shocking you with complete map replacements.
Pros: Smoother target transitions, often more stable, works well in continuous control
Cons: Adds one more hyperparameter (τ) to tune
Which to Choose?
Hard updates work well for discrete action spaces (classic DQN). Soft updates excel in continuous control and actor-critic methods, where smooth value estimates matter more.
Key Takeaway: Hard updates synchronize networks periodically with full weight copying, while soft (Polyak) updates blend networks continuously using an exponential moving average—choose based on your problem domain and required stability.