Understanding the Probability Ratio
What you'll learn: How the probability ratio π_θ(a|s)/π_θ_old(a|s) quantifies policy change in PPO.
What Is the Probability Ratio?
In Proximal Policy Optimization, we need to measure how much our policy has changed after an update. The probability ratio (often denoted as r_t(θ) or just r) is our measurement tool:
r_t(θ) = π_θ(a|s) / π_θ_old(a|s)
This compares the probability of taking action a in state s under the new policy (π_θ) versus the old policy (π_θ_old).
Why This Ratio Matters
Think of the ratio as a "change detector":
- r = 1.0: The new and old policies assign the same probability to this action (no change)
- r > 1.0: The new policy makes this action more likely than before (e.g., r=2.0 means twice as likely)
- r < 1.0: The new policy makes this action less likely (e.g., r=0.5 means half as likely)
Analogy: A Menu Recommendation System
Imagine a restaurant recommendation app. Yesterday's version (old policy) suggested pizza with 20% probability. Today's version (new policy) suggests it with 40% probability. The ratio is 40%/20% = 2.0, telling you the new system is twice as enthusiastic about pizza.
If the ratio were 10.0, you'd know something changed dramatically—and that might be risky! PPO uses this ratio to prevent overly aggressive policy updates.
The Foundation for Clipping
This ratio is the input to PPO's clipped surrogate objective. By monitoring how far r deviates from 1.0, PPO can constrain policy updates to stay within a trust region—achieving TRPO-like stability without expensive computation.
Key Takeaway: The probability ratio π_θ(a|s)/π_θ_old(a|s) quantifies how much the policy's action preferences have shifted, serving as the foundation for PPO's constrained optimization.