Action Space Normalization and Scaling
What you'll learn: How to transform continuous action outputs from neural networks into valid, bounded ranges that your environment expects.
The Core Challenge
When your actor network outputs continuous actions, it produces raw values that might be unbounded (e.g., any real number). But most environments require actions within specific ranges—like steering angles between [-1, 1] or joint torques between [-2, 5]. You need to map network outputs to these valid ranges reliably.
Three Key Techniques
1. Tanh Squashing
Apply the tanh activation to your network's final layer, which naturally bounds outputs to (-1, 1). Think of it like a soft limiter that gradually compresses extreme values.
2. Action Scaling
Once you have bounded outputs (say, from tanh), scale them to match your environment's range. If your environment needs actions in [a_min, a_max]:
scaled_action = a_min + (tanh_output + 1) * (a_max - a_min) / 2
This linearly transforms [-1, 1] to [a_min, a_max].
3. Denormalization
The reverse process: when storing actions in replay buffers or computing gradients, you sometimes need to convert environment actions back to the network's natural scale. This ensures consistent learning signals.
Why This Matters
Without proper normalization, your policy might output invalid actions that get clipped harshly by the environment, breaking the gradient flow. Worse, different action dimensions with vastly different scales (position vs velocity) can destabilize learning—the network struggles to learn appropriate magnitudes for each dimension simultaneously.
Key Takeaway: Action space normalization uses techniques like tanh squashing and linear scaling to reliably map unbounded network outputs into environment-compatible ranges, ensuring valid actions and stable gradient flow during training.