Exploration in Continuous Action Spaces
What you'll learn: How agents discover new behaviors when actions are continuous values rather than discrete choices.
The Exploration Challenge
When your action space is continuous (like steering angles from -30° to +30°, or motor torques), you can't simply use epsilon-greedy to randomly pick from a list. You need strategies that add structured randomness to continuous outputs while still allowing the agent to learn effective policies.
Core Exploration Strategies
Ornstein-Uhlenbeck Noise
Ornstein-Uhlenbeck (OU) noise is the most common approach for algorithms like DDPG. Unlike pure random noise, OU noise is temporally correlated—it has momentum. Think of it like a drunk sailor walking: each step is random, but consecutive steps tend to be in similar directions. This produces smoother, more realistic exploration in physical control tasks.
The noise gradually pulls back toward zero (mean reversion), preventing actions from drifting too far off. It's particularly effective for environments with inertia, like robotic arms or vehicles.
Parameter Space Noise
Instead of adding noise to actions, parameter space noise adds noise directly to your policy network's weights. This creates consistent behavioral changes throughout an entire episode. It's like giving your agent a slightly different "personality" each episode, rather than twitchy moment-to-moment randomness. This often leads to more diverse exploration strategies.
Simple Gaussian Noise
The simplest approach: add independent Gaussian noise to each action dimension. While less sophisticated than OU noise, it's easier to implement and can work well with proper tuning of the noise standard deviation (which typically decays during training).
Key Takeaway: Continuous action exploration requires adding structured noise—either to actions (OU noise, Gaussian noise) or to policy parameters—to balance discovery of new behaviors with exploitation of learned policies.