Course contentsShow
← Machine Learning and Deep Learning
Lesson 2293 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

The TRPO Objective Function

Formulating TRPO as a constrained optimization: maximize expected advantage subject to KL constraint.

The TRPO Objective Function

What you'll learn: How TRPO frames policy improvement as a constrained optimization problem that balances performance gains with safety.

The Core Problem

In standard policy gradient methods, taking too large a step can catastrophically destroy a policy's performance. TRPO (Trust Region Policy Optimization) solves this by explicitly limiting how much the policy can change in a single update.

Formulating TRPO as Constrained Optimization

TRPO transforms the policy improvement problem into:

Maximize: Expected advantage (how much better actions are than average)
Subject to: KL divergence constraint (policy can't change too much)

Mathematically, this looks like:

maximize: E[A^π(s,a)]
subject to: KL(π_old || π_new) ≤ δ

Where:

  • E[A^π(s,a)] is the expected advantage under the new policy
  • KL(π_old || π_new) measures how different the new policy is from the old one
  • δ is the trust region size (a small threshold, like 0.01)

Why This Matters

Think of it like walking in fog: you want to move toward higher ground (better performance), but you can only trust what's nearby. The KL constraint defines your "trust region"—the area where you're confident the advantage estimate is reliable.

By keeping policy updates within this trust region, TRPO guarantees monotonic improvement: each update makes the policy better (or at least no worse), avoiding the destructive large steps that plague vanilla policy gradients.

Key Takeaway: TRPO's objective function maximizes expected advantage while constraining the KL divergence between old and new policies, ensuring safe, reliable policy improvements within a trusted local region.