PPO Hyperparameters and Tuning
What you'll learn: You'll master the key hyperparameters that control PPO's performance and understand how to tune them for stable, efficient learning.
Core Idea
PPO has several critical hyperparameters that dramatically affect training stability, sample efficiency, and final performance. Unlike simpler algorithms, PPO's power comes from balancing exploration, policy stability, and learning speed—all controlled by these knobs.
The Essential Hyperparameters
Learning Rate
Controls how aggressively the policy and value networks update. Typical range: 1e-4 to 3e-4. Too high causes instability; too low wastes samples. Often decayed over training.
Batch Size and Number of Epochs
PPO collects a batch of experience, then runs multiple optimization epochs over it. Common batch size: 2048-4096 steps. Epochs: 3-10. More epochs extract more learning per sample but risk overfitting to old data.
GAE Lambda (λ)
From Generalized Advantage Estimation—balances bias vs. variance in advantage calculations. Range: 0.9 to 0.99. Higher values (like 0.95) use longer-horizon returns, reducing bias but increasing variance.
Clip Range (ε)
The clip parameter in the surrogate objective (typically 0.1 to 0.3). Smaller values mean more conservative updates and greater stability, but slower learning.
Entropy Coefficient
Encourages exploration by penalizing overly deterministic policies. Range: 0.0 to 0.01. Higher values maintain exploration longer but may prevent convergence to optimal behavior.
Tuning Strategy
Start with standard values: learning rate 3e-4, clip range 0.2, GAE lambda 0.95, entropy 0.01, 10 epochs per batch. Adjust learning rate first if training is unstable. Increase entropy if the policy converges prematurely. Reduce clip range if policy updates are too aggressive.
Key Takeaway: PPO's hyperparameters control the trade-off between sample efficiency and training stability—learning rate and clip range are most critical, while GAE lambda and entropy coefficient fine-tune variance and exploration respectively.