Extracting Optimal Policies from V* and Q*
What you'll learn: How to derive the best actions to take in any state once you've computed the optimal value functions V* or Q*.
The Core Idea
Once you've solved for the optimal value functions—either V* (optimal state values) or Q* (optimal action values)—you still need to know what to actually do in each state. This is where policy extraction comes in: converting your knowledge of optimal values into concrete action choices.
The process differs slightly depending on which value function you have:
Extracting Policy from Q*
This is the simpler case. Since Q*(s, a) tells you the value of taking action a in state s, you simply choose the action with the highest Q-value:
π(s) = argmax_a Q(s, a)**
In plain English: "In state s, pick whichever action has the maximum Q* value."
Extracting Policy from V*
This requires one extra step. You need to look ahead one step using the transition dynamics and reward function:
π(s) = argmax_a [R(s, a) + γ Σ P(s'|s,a) V(s')]**
In other words: "Pick the action that maximizes immediate reward plus the discounted value of where you'll likely end up."
Analogy
Think of Q* as a restaurant menu with prices already calculated for each dish—you just pick the best value. V* is like knowing the quality of neighborhoods but needing to calculate travel costs and time to figure out which route gets you to the best destination.
Key Takeaway: Optimal policies are extracted by greedily selecting actions with maximum Q* values (direct lookup) or by one-step lookahead combining rewards and V* values (requires transition model).