Course contentsShow
← Machine Learning and Deep Learning
Lesson 2155 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Extracting Optimal Policies from V* and Q*

Demonstrate how to derive the optimal policy by selecting actions that maximize Q* or lead to V*.

Extracting Optimal Policies from V* and Q*

What you'll learn: How to derive the best actions to take in any state once you've computed the optimal value functions V* or Q*.

The Core Idea

Once you've solved for the optimal value functions—either V* (optimal state values) or Q* (optimal action values)—you still need to know what to actually do in each state. This is where policy extraction comes in: converting your knowledge of optimal values into concrete action choices.

The process differs slightly depending on which value function you have:

Extracting Policy from Q*

This is the simpler case. Since Q*(s, a) tells you the value of taking action a in state s, you simply choose the action with the highest Q-value:

π(s) = argmax_a Q(s, a)**

In plain English: "In state s, pick whichever action has the maximum Q* value."

Extracting Policy from V*

This requires one extra step. You need to look ahead one step using the transition dynamics and reward function:

π(s) = argmax_a [R(s, a) + γ Σ P(s'|s,a) V(s')]**

In other words: "Pick the action that maximizes immediate reward plus the discounted value of where you'll likely end up."

Analogy

Think of Q* as a restaurant menu with prices already calculated for each dish—you just pick the best value. V* is like knowing the quality of neighborhoods but needing to calculate travel costs and time to figure out which route gets you to the best destination.

Key Takeaway: Optimal policies are extracted by greedily selecting actions with maximum Q* values (direct lookup) or by one-step lookahead combining rewards and V* values (requires transition model).