Course contentsShow
← Machine Learning and Deep Learning
Lesson 2327 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Action Space Normalization and Scaling

Techniques for handling bounded vs unbounded actions: tanh squashing, action scaling, and denormalization.

Action Space Normalization and Scaling

What you'll learn: How to transform continuous action outputs from neural networks into valid, bounded ranges that your environment expects.

The Core Challenge

When your actor network outputs continuous actions, it produces raw values that might be unbounded (e.g., any real number). But most environments require actions within specific ranges—like steering angles between [-1, 1] or joint torques between [-2, 5]. You need to map network outputs to these valid ranges reliably.

Three Key Techniques

1. Tanh Squashing

Apply the tanh activation to your network's final layer, which naturally bounds outputs to (-1, 1). Think of it like a soft limiter that gradually compresses extreme values.

2. Action Scaling

Once you have bounded outputs (say, from tanh), scale them to match your environment's range. If your environment needs actions in [a_min, a_max]:

scaled_action = a_min + (tanh_output + 1) * (a_max - a_min) / 2

This linearly transforms [-1, 1] to [a_min, a_max].

3. Denormalization

The reverse process: when storing actions in replay buffers or computing gradients, you sometimes need to convert environment actions back to the network's natural scale. This ensures consistent learning signals.

Why This Matters

Without proper normalization, your policy might output invalid actions that get clipped harshly by the environment, breaking the gradient flow. Worse, different action dimensions with vastly different scales (position vs velocity) can destabilize learning—the network struggles to learn appropriate magnitudes for each dimension simultaneously.

Key Takeaway: Action space normalization uses techniques like tanh squashing and linear scaling to reliably map unbounded network outputs into environment-compatible ranges, ensuring valid actions and stable gradient flow during training.