Policy Improvement Theorem
What you'll learn: How the policy improvement theorem mathematically guarantees that acting greedily with respect to a value function produces a better (or at least as good) policy.
The Core Idea
After evaluating a policy to get its value function V^π, you might wonder: "Can I do better?" The policy improvement theorem answers with a resounding yes—and tells you exactly how.
The theorem states: if you create a new policy π' by acting greedily with respect to the current policy's action-value function Q^π (choosing actions that maximize Q^π at each state), then π' will be at least as good as π, and usually strictly better.
Mathematically, if for all states s:
π'(s) = argmax_a Q^π(s,a)
Then V^π'(s) ≥ V^π(s) for all states s.
Why This Works
Think of it like climbing a hill: your current policy tells you where you are (giving you V^π). The Q-function shows you which direction leads upward from each spot. By always choosing the steepest upward path (acting greedily), you're guaranteed not to go downhill.
The proof relies on the Bellman equations: choosing the action that maximizes Q^π means you're getting at least as much value in that first step, and you continue optimally from there.
The Foundation of Policy Iteration
This theorem is why policy iteration works: alternate between policy evaluation (computing V^π) and policy improvement (acting greedily). Each improvement step provably makes the policy better, until you reach the optimal policy π* where greedy improvement changes nothing.
Key Takeaway: The policy improvement theorem guarantees that acting greedily with respect to Q^π produces a policy that is provably better or equal, forming the mathematical foundation for iterative policy optimization algorithms.