Course contentsShow
← Machine Learning and Deep Learning
Lesson 2215 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Reward Clipping and Normalization

Scaling rewards to a standard range to improve learning stability across different environments.

Reward Clipping and Normalization

What you'll learn: How to stabilize deep RL training by scaling rewards to a consistent range across different environments.

The Problem: Wild Reward Scales

When training Deep Q-Networks across different Atari games (or any diverse set of environments), you'll encounter a serious challenge: reward scales vary wildly. One game might give +1 for each point scored, reaching thousands per episode, while another gives +10 or +100 for rare events. These inconsistent magnitudes destabilize learning because the same neural network architecture must handle drastically different value ranges.

Reward Clipping

Reward clipping bounds all rewards to a fixed range, typically [-1, +1]. Any positive reward becomes +1, any negative becomes -1, and zero stays zero:

def clip_reward(reward):
    if reward > 0:
        return 1.0
    elif reward < 0:
        return -1.0
    else:
        return 0.0

This was famously used in the original DQN paper. The benefit? Your network learns relative value relationships without being overwhelmed by magnitude differences. The downside? You lose reward granularity—winning by 10 points looks identical to winning by 100.

Reward Normalization

Reward normalization scales rewards using running statistics (mean and standard deviation):

normalized_reward = (reward - mean) / (std + epsilon)

This preserves relative differences while standardizing the distribution. It's gentler than clipping but requires tracking statistics across training.

When to Use Which

Clipping works well when you care about "did something good/bad happen?" rather than "how good?" It's simple and prevents outliers from domaging gradients. Normalization preserves more information but adds complexity.

Key Takeaway: Reward clipping and normalization transform diverse reward scales into a consistent range, preventing large rewards from dominating gradient updates and enabling a single network architecture to learn effectively across multiple environments.