The Bellman Optimality Equation for V*
What you'll learn: How to express the optimal state value function V* as a recursive equation that guides finding the best possible policy.
The Core Idea
You've learned that V*(s) represents the maximum value achievable from state s under the best possible policy. But how do we actually compute it?
The Bellman Optimality Equation for V* tells us that the optimal value of being in state s equals the best immediate reward plus discounted future value we can get by choosing the best action:
V(s) = max over all actions a of [ R(s,a) + γ · Σ P(s'|s,a) · V(s') ]**
In plain English: "The best value from state s is found by picking whichever action gives you the highest combination of immediate reward and expected future optimal value."
Why "Max" Instead of Expectation?
Recall the Bellman Expectation Equation averaged over actions according to some policy π. Here, we're not following a fixed policy—we're choosing the best action at every state. The "max" operator captures this optimization: we look at all possible actions and pick the one that maximizes our return.
The Recursive Structure
This equation is self-referential: V*(s) is defined in terms of V*(s') for successor states. This recursion means:
- At terminal states, V* is just the immediate reward
- Working backward, each state's optimal value depends on its successors' optimal values
- This structure enables dynamic programming solutions like Value Iteration
Key Takeaway: The Bellman Optimality Equation for V* states that the optimal value of a state equals the maximum expected return over all actions, creating a recursive condition that defines optimality throughout the MDP.