Action-Value Functions: Q-Functions
What you'll learn: How Q-functions extend value functions by evaluating the expected return from taking a specific action in a specific state, then following a policy thereafter.
From State Value to Action Value
In the previous lesson, you learned about state-value functions V^π(s), which tell us how good it is to be in a state under policy π. But what if we want to know how good it is to take a specific action from that state?
That's where action-value functions, commonly called Q-functions, come in.
What is a Q-Function?
The Q-function for policy π is defined as:
Q^π(s, a) = E[G_t | S_t=s, A_t=a]
This reads: "The Q-value of taking action a in state s under policy π is the expected return G_t (cumulative discounted reward), given that we start in state s, take action a, and then follow policy π for all subsequent steps."
Why Q-Functions Matter
Think of Q-functions as action ratings. While V(s) tells you "this state is worth X," Q(s, a) tells you "taking this particular action from this state is worth Y."
This granularity is crucial for:
- Action selection: Compare Q-values for different actions to choose the best one
- Policy improvement: Identify which actions are better than what your current policy suggests
- Learning algorithms: Many RL algorithms (like Q-learning) directly learn Q-functions rather than value functions
Relationship to State Value
The state-value function is actually the expectation over Q-values:
V^π(s) = E_{a~π}[Q^π(s, a)]
If you follow policy π, your expected value equals the weighted average of all action-values from that state.
Key Takeaway: Q-functions Q^π(s,a) estimate the expected return from taking a specific action in a state, providing action-level granularity essential for decision-making in reinforcement learning.