Course contentsShow
← Machine Learning and Deep Learning
Lesson 2320 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Exploration in Continuous Action Spaces

Ornstein-Uhlenbeck noise, parameter space noise, and other exploration strategies for continuous control.

Exploration in Continuous Action Spaces

What you'll learn: How agents discover new behaviors when actions are continuous values rather than discrete choices.

The Exploration Challenge

When your action space is continuous (like steering angles from -30° to +30°, or motor torques), you can't simply use epsilon-greedy to randomly pick from a list. You need strategies that add structured randomness to continuous outputs while still allowing the agent to learn effective policies.

Core Exploration Strategies

Ornstein-Uhlenbeck Noise

Ornstein-Uhlenbeck (OU) noise is the most common approach for algorithms like DDPG. Unlike pure random noise, OU noise is temporally correlated—it has momentum. Think of it like a drunk sailor walking: each step is random, but consecutive steps tend to be in similar directions. This produces smoother, more realistic exploration in physical control tasks.

The noise gradually pulls back toward zero (mean reversion), preventing actions from drifting too far off. It's particularly effective for environments with inertia, like robotic arms or vehicles.

Parameter Space Noise

Instead of adding noise to actions, parameter space noise adds noise directly to your policy network's weights. This creates consistent behavioral changes throughout an entire episode. It's like giving your agent a slightly different "personality" each episode, rather than twitchy moment-to-moment randomness. This often leads to more diverse exploration strategies.

Simple Gaussian Noise

The simplest approach: add independent Gaussian noise to each action dimension. While less sophisticated than OU noise, it's easier to implement and can work well with proper tuning of the noise standard deviation (which typically decays during training).

Key Takeaway: Continuous action exploration requires adding structured noise—either to actions (OU noise, Gaussian noise) or to policy parameters—to balance discovery of new behaviors with exploitation of learned policies.