Implementing PPO in PyTorch
What you'll learn: How to translate PPO's clipped surrogate objective into working PyTorch code with actor-critic networks and a complete training loop.
Building the Actor-Critic Networks
PPO requires two neural networks that can share a backbone. The actor outputs a probability distribution over actions (policy), while the critic estimates state values. In PyTorch, you'll typically define separate heads after shared convolutional or dense layers:
class ActorCritic(nn.Module):
def __init__(self, state_dim, action_dim):
super().__init__()
self.shared = nn.Sequential(
nn.Linear(state_dim, 64),
nn.ReLU()
)
self.actor = nn.Linear(64, action_dim)
self.critic = nn.Linear(64, 1)
Computing the Clipped Objective
The heart of PPO is computing the probability ratio r_t = π_new(a|s) / π_old(a|s) and applying the clip. You'll need to:
- Store old log probabilities during rollout collection
- Compute new log probabilities from the current policy
- Calculate the ratio using
torch.exp(new_log_prob - old_log_prob) - Apply the clip:
torch.min(ratio * advantage, torch.clamp(ratio, 1-ε, 1+ε) * advantage)
The Training Loop
PPO runs multiple epochs over collected trajectories. For each mini-batch:
- Compute advantages using GAE (already covered)
- Calculate both actor loss (clipped objective) and critic loss (MSE between predicted and target values)
- Add entropy bonus for exploration
- Backpropagate the combined loss
- Repeat for several epochs before collecting new data
Think of it like refining pottery: you collect clay (trajectories), then carefully shape it multiple times (epochs) without distorting too much (clipping).
Key Takeaway: PPO implementation combines actor-critic networks, careful probability ratio computation with clipping, and multiple optimization passes over fixed trajectory batches—balancing sample efficiency with stable, conservative policy updates.