Course contentsShow
← Machine Learning and Deep Learning
Lesson 2271 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Handling Continuous Action Spaces

Using Gaussian policies and reparameterization for REINFORCE in continuous control tasks.

Handling Continuous Action Spaces

What you'll learn: How to apply REINFORCE to continuous control tasks using Gaussian policies and the reparameterization trick.

The Challenge of Continuous Actions

In many real-world problems—robot arm control, autonomous driving, or adjusting throttle—actions aren't discrete choices but continuous values. Instead of selecting "left" or "right," you need to output a precise steering angle like -12.7°. Your policy must now output a distribution over continuous values, not just probabilities over discrete actions.

Gaussian Policies

The most common approach: your neural network outputs the mean (μ) and standard deviation (σ) of a Gaussian (normal) distribution. Think of it like aiming at a dartboard—μ is where you're aiming, and σ is how much your hand shakes.

mu = policy_network(state)  # Mean action
sigma = torch.exp(log_std)  # Standard deviation (ensure positive)

# Sample action from N(μ, σ)
action = torch.normal(mu, sigma)

Reparameterization for REINFORCE

To compute gradients through this sampling process, we use the reparameterization trick: instead of sampling directly from N(μ, σ), we sample noise ε from N(0, 1) and compute:

action = μ + σ × ε

This separates the randomness from the parameters, allowing gradients to flow back through μ and σ. You can then compute the log-probability of the action under this Gaussian and use it in your REINFORCE update, exactly as before:

dist = torch.distributions.Normal(mu, sigma)
log_prob = dist.log_prob(action).sum()  # For multi-dimensional actions
loss = -log_prob * return_value

Key Takeaway: Continuous action spaces require sampling from parameterized distributions (typically Gaussian); the reparameterization trick enables gradient-based learning by making the sampling operation differentiable with respect to policy parameters.