Course contentsShow
← Machine Learning and Deep Learning
Lesson 2161 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Policy Improvement Theorem

Study the policy improvement theorem that guarantees acting greedily with respect to a value function improves the policy.

Policy Improvement Theorem

What you'll learn: How the policy improvement theorem mathematically guarantees that acting greedily with respect to a value function produces a better (or at least as good) policy.

The Core Idea

After evaluating a policy to get its value function V^π, you might wonder: "Can I do better?" The policy improvement theorem answers with a resounding yes—and tells you exactly how.

The theorem states: if you create a new policy π' by acting greedily with respect to the current policy's action-value function Q^π (choosing actions that maximize Q^π at each state), then π' will be at least as good as π, and usually strictly better.

Mathematically, if for all states s:

π'(s) = argmax_a Q^π(s,a)

Then V^π'(s) ≥ V^π(s) for all states s.

Why This Works

Think of it like climbing a hill: your current policy tells you where you are (giving you V^π). The Q-function shows you which direction leads upward from each spot. By always choosing the steepest upward path (acting greedily), you're guaranteed not to go downhill.

The proof relies on the Bellman equations: choosing the action that maximizes Q^π means you're getting at least as much value in that first step, and you continue optimally from there.

The Foundation of Policy Iteration

This theorem is why policy iteration works: alternate between policy evaluation (computing V^π) and policy improvement (acting greedily). Each improvement step provably makes the policy better, until you reach the optimal policy π* where greedy improvement changes nothing.

Key Takeaway: The policy improvement theorem guarantees that acting greedily with respect to Q^π produces a policy that is provably better or equal, forming the mathematical foundation for iterative policy optimization algorithms.