Course contentsShow
← Machine Learning and Deep Learning
Lesson 2143 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Action-Value Functions: Q-Functions

Expected return from state-action pairs: Q^π(s,a) = E[G_t | S_t=s, A_t=a].

Action-Value Functions: Q-Functions

What you'll learn: How Q-functions extend value functions by evaluating the expected return from taking a specific action in a specific state, then following a policy thereafter.

From State Value to Action Value

In the previous lesson, you learned about state-value functions V^π(s), which tell us how good it is to be in a state under policy π. But what if we want to know how good it is to take a specific action from that state?

That's where action-value functions, commonly called Q-functions, come in.

What is a Q-Function?

The Q-function for policy π is defined as:

Q^π(s, a) = E[G_t | S_t=s, A_t=a]

This reads: "The Q-value of taking action a in state s under policy π is the expected return G_t (cumulative discounted reward), given that we start in state s, take action a, and then follow policy π for all subsequent steps."

Why Q-Functions Matter

Think of Q-functions as action ratings. While V(s) tells you "this state is worth X," Q(s, a) tells you "taking this particular action from this state is worth Y."

This granularity is crucial for:

  • Action selection: Compare Q-values for different actions to choose the best one
  • Policy improvement: Identify which actions are better than what your current policy suggests
  • Learning algorithms: Many RL algorithms (like Q-learning) directly learn Q-functions rather than value functions

Relationship to State Value

The state-value function is actually the expectation over Q-values:

V^π(s) = E_{a~π}[Q^π(s, a)]

If you follow policy π, your expected value equals the weighted average of all action-values from that state.

Key Takeaway: Q-functions Q^π(s,a) estimate the expected return from taking a specific action in a state, providing action-level granularity essential for decision-making in reinforcement learning.