Reward Clipping and Normalization
What you'll learn: How to stabilize deep RL training by scaling rewards to a consistent range across different environments.
The Problem: Wild Reward Scales
When training Deep Q-Networks across different Atari games (or any diverse set of environments), you'll encounter a serious challenge: reward scales vary wildly. One game might give +1 for each point scored, reaching thousands per episode, while another gives +10 or +100 for rare events. These inconsistent magnitudes destabilize learning because the same neural network architecture must handle drastically different value ranges.
Reward Clipping
Reward clipping bounds all rewards to a fixed range, typically [-1, +1]. Any positive reward becomes +1, any negative becomes -1, and zero stays zero:
def clip_reward(reward):
if reward > 0:
return 1.0
elif reward < 0:
return -1.0
else:
return 0.0
This was famously used in the original DQN paper. The benefit? Your network learns relative value relationships without being overwhelmed by magnitude differences. The downside? You lose reward granularity—winning by 10 points looks identical to winning by 100.
Reward Normalization
Reward normalization scales rewards using running statistics (mean and standard deviation):
normalized_reward = (reward - mean) / (std + epsilon)
This preserves relative differences while standardizing the distribution. It's gentler than clipping but requires tracking statistics across training.
When to Use Which
Clipping works well when you care about "did something good/bad happen?" rather than "how good?" It's simple and prevents outliers from domaging gradients. Normalization preserves more information but adds complexity.
Key Takeaway: Reward clipping and normalization transform diverse reward scales into a consistent range, preventing large rewards from dominating gradient updates and enabling a single network architecture to learn effectively across multiple environments.