Course contentsShow
← Machine Learning and Deep Learning
Lesson 2154 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Relationship Between V* and Q*

Show how optimal state values and action values relate: V*(s) = max_a Q*(s,a).

Relationship Between V* and Q*

What you'll learn: How the optimal state value and optimal action-value functions connect through a simple maximization.

The Core Connection

In reinforcement learning, the optimal state value V*(s) and optimal action-value Q*(s,a) are two sides of the same coin. They're related by a beautifully simple equation:

V(s) = max_a Q(s,a)**

This says: The value of being in the best possible situation at state s equals the value of taking the best action available from that state.

Understanding the Relationship

Think of it like choosing a restaurant in a new city:

  • Q(s,a)* is the expected satisfaction you'd get from choosing each specific restaurant (action a) given where you are (state s)
  • V(s)* is the satisfaction you'll actually get — because you'll obviously pick the best restaurant available

When you're acting optimally, you always choose the action that maximizes Q. So the value of being in state s under an optimal policy is simply the maximum Q-value among all possible actions.

Why This Matters

This relationship is fundamental because:

  1. It shows you can derive one function from the other
  2. If you know all Q*(s,a) values, finding the optimal action is trivial: just pick argmax_a Q*(s,a)
  3. It connects the two Bellman optimality equations you've learned

The equation bridges state-centric thinking (V*) with action-centric thinking (Q*), giving you flexibility in how you approach solving MDPs.

Key Takeaway: The optimal state value V*(s) equals the maximum action-value Q*(s,a) over all actions — you get V* by always choosing the best action's Q-value.