The Clipped Surrogate Objective
What you'll learn: How PPO cleverly limits policy updates by clipping the probability ratio, achieving TRPO's safety without expensive second-order calculations.
The Core Innovation
Remember TRPO's goal? Keep policy updates safe by constraining how much the new policy can differ from the old one using KL divergence. The problem? Computing that constraint requires the Fisher Information Matrix and conjugate gradients—computationally expensive operations.
PPO's brilliant insight: instead of explicitly constraining updates with KL divergence, clip the objective function itself to prevent destructively large policy changes.
How Clipping Works
Recall the surrogate objective from TRPO, which uses the probability ratio:
r(θ) = π_θ(a|s) / π_θ_old(a|s)
PPO clips this ratio to stay within [1-ε, 1+ε] (typically ε=0.2):
L^CLIP(θ) = E[min(r(θ)·A, clip(r(θ), 1-ε, 1+ε)·A)]
The intuition: If the advantage A is positive (good action), we want to increase its probability—but not too much. The clip prevents r(θ) from exceeding 1.2×. If A is negative (bad action), we decrease probability but cap the reduction at 0.8×.
The min() operation is key: it takes the pessimistic bound. If increasing probability would make r(θ) too large, the clipped version kicks in and flattens the objective. Further updates in that direction provide no additional reward—the gradient naturally stops pushing.
Why This Works
This simple clip approximates a trust region without computing Hessians. Once the ratio hits the boundary, the objective becomes flat in that direction, effectively stopping further updates. It's like guardrails on a highway—cheap to install, effective at preventing dangerous deviations.
Key Takeaway: PPO's clipped surrogate objective constrains policy updates by capping the probability ratio, achieving TRPO-like safety with first-order optimization—making it dramatically more practical and efficient.