Course contentsShow
← Machine Learning and Deep Learning
Lesson 2311 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Implementing PPO in PyTorch

Building actor-critic networks, computing clipped objectives, and implementing the training loop.

Implementing PPO in PyTorch

What you'll learn: How to translate PPO's clipped surrogate objective into working PyTorch code with actor-critic networks and a complete training loop.

Building the Actor-Critic Networks

PPO requires two neural networks that can share a backbone. The actor outputs a probability distribution over actions (policy), while the critic estimates state values. In PyTorch, you'll typically define separate heads after shared convolutional or dense layers:

class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.shared = nn.Sequential(
            nn.Linear(state_dim, 64),
            nn.ReLU()
        )
        self.actor = nn.Linear(64, action_dim)
        self.critic = nn.Linear(64, 1)

Computing the Clipped Objective

The heart of PPO is computing the probability ratio r_t = π_new(a|s) / π_old(a|s) and applying the clip. You'll need to:

  1. Store old log probabilities during rollout collection
  2. Compute new log probabilities from the current policy
  3. Calculate the ratio using torch.exp(new_log_prob - old_log_prob)
  4. Apply the clip: torch.min(ratio * advantage, torch.clamp(ratio, 1-ε, 1+ε) * advantage)

The Training Loop

PPO runs multiple epochs over collected trajectories. For each mini-batch:

  • Compute advantages using GAE (already covered)
  • Calculate both actor loss (clipped objective) and critic loss (MSE between predicted and target values)
  • Add entropy bonus for exploration
  • Backpropagate the combined loss
  • Repeat for several epochs before collecting new data

Think of it like refining pottery: you collect clay (trajectories), then carefully shape it multiple times (epochs) without distorting too much (clipping).

Key Takeaway: PPO implementation combines actor-critic networks, careful probability ratio computation with clipping, and multiple optimization passes over fixed trajectory batches—balancing sample efficiency with stable, conservative policy updates.