Optimal Value Functions: V* and Q*
What you'll learn: How to define the "best possible" value functions that represent the maximum achievable expected return in an MDP.
The Core Idea
In reinforcement learning, we've seen that different policies lead to different value functions. But what if we could find the best policy possible? The optimal state-value function V(s)* tells us the maximum expected return we can achieve starting from state s when we follow the absolute best policy. Similarly, the optimal action-value function Q(s, a)* gives us the maximum expected return from taking action a in state s and then following the best policy afterward.
Think of it like this: if you're playing chess, V*(s) is the best outcome you can guarantee from any board position if you play perfectly from that point forward. Q*(s, a) is the best outcome from making a specific move and then playing perfectly afterward.
Mathematical Definition
The optimal value functions are defined as the maximum over all possible policies:
- V(s) = max_π V^π(s)* — the best possible state value
- Q(s, a) = max_π Q^π(s, a)* — the best possible action value
These functions satisfy the Bellman Optimality Equations, which state that the optimal value equals the reward plus the discounted optimal value of the best next state.
Why This Matters
Once we know V* or Q*, finding the optimal policy becomes straightforward: just pick actions that lead to states with maximum optimal value. The entire goal of many RL algorithms is to estimate or compute these optimal value functions.
Key Takeaway: V* and Q* represent the ceiling of performance in an MDP—the best expected return achievable from any state or state-action pair under perfect decision-making.