Generalized Policy Iteration Framework
What you'll learn: You'll understand how GPI provides a unifying view of reinforcement learning algorithms as the interplay between evaluation and improvement.
The Core Pattern
Throughout your study of policy iteration and value iteration, you've seen a recurring dance: we evaluate how good our current policy is, then we improve it based on what we learned. Generalized Policy Iteration (GPI) is the recognition that this back-and-forth pattern is the fundamental heartbeat of most RL algorithms.
GPI isn't a specific algorithm—it's a conceptual framework that describes how nearly all RL methods work. Think of it as two processes running in parallel:
- Policy Evaluation: Making the value function consistent with the current policy (answering "how good is this policy?")
- Policy Improvement: Making the policy greedy with respect to the current value function (answering "can we do better?")
The Interleaving Dance
The beauty of GPI is flexibility. You don't need to complete either process fully before switching:
- Policy Iteration: Evaluation runs to convergence, then one full improvement step
- Value Iteration: Just one evaluation backup per state, then immediate improvement
- Asynchronous methods: Update states in any order, mixing evaluation and improvement freely
The two processes push against each other—evaluation tries to match the current policy, while improvement changes that policy. Yet together, they converge toward optimality, like two opposing forces finding equilibrium.
Why It Matters
GPI reveals that all the algorithms you've learned (and many you'll encounter) are variations on the same theme. They differ only in how much evaluation happens before improvement, and which states get updated when.
Key Takeaway: Generalized Policy Iteration is the unifying framework showing that RL algorithms converge to optimal policies through continuous interleaving of policy evaluation and policy improvement, even when neither process completes fully before switching.