Course contentsShow
← Machine Learning and Deep Learning
Lesson 2224 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Target Network Update Strategies

Hard updates versus soft (Polyak) updates for synchronizing target networks with online networks.

Target Network Update Strategies

What you'll learn: How to synchronize your target network with the online network using hard updates versus soft (Polyak) updates in Deep Q-Networks.

The Core Problem

You already know that DQN uses a separate target network to stabilize training by providing consistent Q-value targets. But how often and by how much should you update this target network? Update it too frequently and you lose stability; update it too rarely and your targets become stale.

Hard Updates (Original DQN Approach)

The simplest strategy: copy all weights from the online network to the target network every C training steps (e.g., every 10,000 steps).

if step % target_update_frequency == 0:
    target_network.load_state_dict(online_network.state_dict())

Analogy: Like replacing your GPS map completely every few hours. You get all new information at once, but nothing changes between updates.

Pros: Simple, proven to work
Cons: Creates discontinuities in target values; targets suddenly "jump"

Soft Updates (Polyak Averaging)

Introduced by DDPG and other algorithms, soft updates blend the networks gradually at every step using parameter τ (tau), typically 0.001–0.01:

# After every training step
for target_param, online_param in zip(target_network.parameters(), 
                                       online_network.parameters()):
    target_param.data.copy_(τ * online_param.data + (1 - τ) * target_param.data)

Analogy: Like your GPS gradually updating the map as you drive, smoothly incorporating new road data rather than shocking you with complete map replacements.

Pros: Smoother target transitions, often more stable, works well in continuous control
Cons: Adds one more hyperparameter (τ) to tune

Which to Choose?

Hard updates work well for discrete action spaces (classic DQN). Soft updates excel in continuous control and actor-critic methods, where smooth value estimates matter more.

Key Takeaway: Hard updates synchronize networks periodically with full weight copying, while soft (Polyak) updates blend networks continuously using an exponential moving average—choose based on your problem domain and required stability.