Course contentsShow
← Machine Learning and Deep Learning
Lesson 2301 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Motivation: Why PPO After TRPO?

Understanding the computational complexity of TRPO and the need for a simpler, more practical algorithm.

Motivation: Why PPO After TRPO?

What You'll Learn

You'll understand why the research community needed a simpler alternative to TRPO despite its strong theoretical foundations.

The Computational Burden of TRPO

TRPO was a breakthrough in safe policy optimization, but it came with a heavy price: computational complexity. Remember those components from previous lessons? The conjugate gradient method to solve the constrained optimization problem, the Fisher Information Matrix computation, and the line search for step size selection all added significant overhead to each policy update.

Think of TRPO like a precision surgical robot: incredibly accurate and safe, but requiring extensive setup, calibration, and computational resources for every single operation. When you're training agents that need millions of policy updates, this complexity becomes a bottleneck.

The Need for Simplicity

While TRPO's constraint on KL divergence elegantly prevented destructive policy updates, actually enforcing this constraint required:

  • Computing second-order derivatives (the Fisher Information Matrix)
  • Running iterative conjugate gradient optimization
  • Performing line searches to satisfy constraints

This made TRPO difficult to implement, slow to run, and hard to scale to large networks or parallel training setups.

Enter PPO's Promise

The motivation for Proximal Policy Optimization (PPO) was simple: can we achieve TRPO's stable, monotonic improvement while using only first-order optimization? Could we replace the complex constraint enforcement with something that works with standard gradient descent?

The research community wanted an algorithm that practitioners could actually implement and tune without specialized optimization expertise, while preserving TRPO's safety guarantees as much as possible.

Key Takeaway: PPO was motivated by TRPO's computational complexity — the need for a practical algorithm that maintains stable policy updates without requiring second-order optimization, conjugate gradients, or complex line searches.