The TRPO Objective Function
What you'll learn: How TRPO frames policy improvement as a constrained optimization problem that balances performance gains with safety.
The Core Problem
In standard policy gradient methods, taking too large a step can catastrophically destroy a policy's performance. TRPO (Trust Region Policy Optimization) solves this by explicitly limiting how much the policy can change in a single update.
Formulating TRPO as Constrained Optimization
TRPO transforms the policy improvement problem into:
Maximize: Expected advantage (how much better actions are than average)
Subject to: KL divergence constraint (policy can't change too much)
Mathematically, this looks like:
maximize: E[A^π(s,a)]
subject to: KL(π_old || π_new) ≤ δ
Where:
E[A^π(s,a)]is the expected advantage under the new policyKL(π_old || π_new)measures how different the new policy is from the old oneδis the trust region size (a small threshold, like 0.01)
Why This Matters
Think of it like walking in fog: you want to move toward higher ground (better performance), but you can only trust what's nearby. The KL constraint defines your "trust region"—the area where you're confident the advantage estimate is reliable.
By keeping policy updates within this trust region, TRPO guarantees monotonic improvement: each update makes the policy better (or at least no worse), avoiding the destructive large steps that plague vanilla policy gradients.
Key Takeaway: TRPO's objective function maximizes expected advantage while constraining the KL divergence between old and new policies, ensuring safe, reliable policy improvements within a trusted local region.