Course contentsShow
← Machine Learning and Deep Learning
Lesson 2316 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Policy Representation for Continuous Actions

Parameterizing policies as Gaussian distributions: mean and variance networks, diagonal vs full covariance matrices.

Policy Representation for Continuous Actions

What You'll Learn

You'll learn how to parameterize continuous action policies as Gaussian distributions using neural networks to output mean and variance.

The Core Idea

When your agent needs to choose continuous actions (like steering angles or motor torques), you can't just output discrete probabilities. Instead, you model the policy as a probability distribution over continuous values—typically a Gaussian (normal) distribution.

Your neural network outputs the parameters of this Gaussian:

Mean Network

The network produces a mean vector μ for each action dimension. This represents the "center" of where the agent wants to act. For a 3D action space, you'd output three mean values.

Variance Network

You also need to specify uncertainty or exploration. The network outputs variance (or standard deviation) parameters σ² that control how "spread out" the distribution is. Higher variance = more exploration; lower variance = more exploitation.

Covariance Matrix Choices

Diagonal Covariance: The simplest approach treats each action dimension independently. Your network outputs one variance per dimension. This means a 3D action space needs 3 variance values. It's computationally cheap but assumes actions don't correlate.

Full Covariance: Models dependencies between action dimensions with a full covariance matrix. If moving the arm left correlates with rotating the wrist, a full matrix captures this. However, you need to output and store many more parameters (for n dimensions: n² values), and ensure the matrix stays valid (positive semi-definite).

Practical Implementation

Most implementations use diagonal covariance for simplicity and stability. The network outputs [μ₁, μ₂, ..., μₙ, σ₁, σ₂, ..., σₙ], then you sample actions from N(μᵢ, σᵢ²) for each dimension independently.

Key Takeaway: Continuous action policies are parameterized as Gaussian distributions where neural networks output mean and variance parameters; diagonal covariance treats dimensions independently (simple, fast), while full covariance models correlations (complex, expressive).