Action-Value Functions (Q-Functions)
What You'll Learn
You'll learn how Q-functions extend value functions by evaluating not just states, but the combination of states and actions together.
From State Value to Action Value
In the previous lesson, you learned about state value functions V(s), which tell us how good it is to be in a particular state. But what if we want to know how good it is to take a specific action in that state?
That's where action-value functions, commonly called Q-functions, come in.
The Core Idea
A Q-function, written as Q(s, a), answers this question:
"If I'm in state s and take action a, what's the expected total return I'll get from that point forward?"
The Q-function evaluates state-action pairs, not just states. This is crucial because different actions in the same state can lead to vastly different outcomes.
Why Q-Functions Matter
Think of navigating a gridworld. Being in a particular cell (state) might be moderately good, but:
- Moving right might lead you toward the goal (high Q-value)
- Moving left might lead you into a wall (low Q-value)
The Q-function captures these distinctions. Once you know Q(s, a) for all actions in a state, choosing the optimal action becomes straightforward: pick the action with the highest Q-value!
Relationship to State Value
The state value V(s) and Q-function are related. The value of a state is simply the Q-value of the best action you could take from that state (under an optimal policy).
Key Takeaway: Q-functions (Q(s,a)) evaluate state-action pairs, telling you the expected return from taking action a in state s. They're essential for action selection in reinforcement learning because they directly compare different choices.