The Surrogate Objective
What you'll learn: How to construct an importance-weighted advantage estimator that enables off-policy-like updates in advanced policy gradient methods.
The Core Problem
In lesson 2293, we defined the TRPO objective using a trust region. But here's the challenge: we want to update our policy using data collected from an old policy. Directly applying the policy gradient would be wrong because the trajectories weren't sampled from our current policy.
Importance Sampling to the Rescue
The surrogate objective solves this by weighting each action's advantage by the probability ratio between new and old policies:
L(θ) = E[ (π_θ(a|s) / π_θ_old(a|s)) * A^π_θ_old(s,a) ]
Think of it like this: if your new policy makes an action more likely than the old one did, and that action had positive advantage, you amplify that signal. If it makes a bad action less likely, you get credit for that improvement too.
Why "Surrogate"?
We call it a "surrogate" because it's not the true objective (expected return under the new policy). Instead, it's a local approximation that's valid when the policies are similar. The importance weight π_θ / π_θ_old corrects for the mismatch between where we sampled data and where we want to improve.
This formulation is what makes TRPO and later PPO so powerful—you can reuse trajectory data for multiple updates instead of throwing it away after one gradient step, dramatically improving sample efficiency.
Key Takeaway: The surrogate objective uses importance weighting to reuse old trajectory data for policy updates, enabling safer, more sample-efficient learning by correcting for the distribution mismatch between old and new policies.