Handling Continuous Action Spaces
What you'll learn: How to apply REINFORCE to continuous control tasks using Gaussian policies and the reparameterization trick.
The Challenge of Continuous Actions
In many real-world problems—robot arm control, autonomous driving, or adjusting throttle—actions aren't discrete choices but continuous values. Instead of selecting "left" or "right," you need to output a precise steering angle like -12.7°. Your policy must now output a distribution over continuous values, not just probabilities over discrete actions.
Gaussian Policies
The most common approach: your neural network outputs the mean (μ) and standard deviation (σ) of a Gaussian (normal) distribution. Think of it like aiming at a dartboard—μ is where you're aiming, and σ is how much your hand shakes.
mu = policy_network(state) # Mean action
sigma = torch.exp(log_std) # Standard deviation (ensure positive)
# Sample action from N(μ, σ)
action = torch.normal(mu, sigma)
Reparameterization for REINFORCE
To compute gradients through this sampling process, we use the reparameterization trick: instead of sampling directly from N(μ, σ), we sample noise ε from N(0, 1) and compute:
action = μ + σ × ε
This separates the randomness from the parameters, allowing gradients to flow back through μ and σ. You can then compute the log-probability of the action under this Gaussian and use it in your REINFORCE update, exactly as before:
dist = torch.distributions.Normal(mu, sigma)
log_prob = dist.log_prob(action).sum() # For multi-dimensional actions
loss = -log_prob * return_value
Key Takeaway: Continuous action spaces require sampling from parameterized distributions (typically Gaussian); the reparameterization trick enables gradient-based learning by making the sampling operation differentiable with respect to policy parameters.