Course contentsShow
← Machine Learning and Deep Learning
Lesson 2312 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

PPO for Continuous and Discrete Actions

Adapting PPO for different action spaces using Gaussian policies or categorical distributions.

PPO for Continuous and Discrete Actions

What You'll Learn

You'll learn how to adapt PPO's implementation for both continuous and discrete action spaces by choosing the appropriate output distribution.

The Core Idea

PPO works seamlessly with different action spaces, but the policy network's output layer and sampling mechanism must match the action type. The clipped objective and advantage estimation remain identical—only how you represent and sample actions changes.

Discrete Actions: Categorical Distribution

For discrete actions (like "move left," "jump," "shoot"), your policy network outputs logits for each possible action. You then:

  1. Apply softmax to get probabilities
  2. Sample from a categorical distribution
  3. Compute log probabilities for the PPO ratio

Think of it like a multiple-choice test: the network assigns confidence scores to each option, and you pick one probabilistically.

# Discrete action sampling
logits = actor_network(state)  # shape: [batch, num_actions]
dist = torch.distributions.Categorical(logits=logits)
action = dist.sample()  # integer from 0 to num_actions-1
log_prob = dist.log_prob(action)

Continuous Actions: Gaussian Policy

For continuous actions (like steering angle or motor torque), your network outputs mean (and optionally standard deviation) parameters. You sample from a Gaussian (Normal) distribution:

# Continuous action sampling
mean = actor_network(state)  # shape: [batch, action_dim]
std = torch.exp(log_std)  # learnable or fixed
dist = torch.distributions.Normal(mean, std)
action = dist.sample()  # continuous values
log_prob = dist.log_prob(action).sum(dim=-1)  # sum across dimensions

The Gaussian policy allows smooth, fine-grained control in continuous environments.

Key Implementation Detail

The PPO ratio π_new(a|s) / π_old(a|s) becomes exp(log_prob_new - log_prob_old) regardless of distribution type—this is why PPO elegantly handles both cases with the same clipping logic.

Key Takeaway: PPO adapts to action spaces through distribution choice—categorical for discrete, Gaussian for continuous—while keeping the core clipped objective unchanged.