Course contentsShow
← Machine Learning and Deep Learning
Lesson 2148 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Action-Value Functions (Q-Functions)

Introduce Q(s,a) functions that evaluate the expected return from taking action a in state s.

Action-Value Functions (Q-Functions)

What You'll Learn

You'll learn how Q-functions extend value functions by evaluating not just states, but the combination of states and actions together.

From State Value to Action Value

In the previous lesson, you learned about state value functions V(s), which tell us how good it is to be in a particular state. But what if we want to know how good it is to take a specific action in that state?

That's where action-value functions, commonly called Q-functions, come in.

The Core Idea

A Q-function, written as Q(s, a), answers this question:

"If I'm in state s and take action a, what's the expected total return I'll get from that point forward?"

The Q-function evaluates state-action pairs, not just states. This is crucial because different actions in the same state can lead to vastly different outcomes.

Why Q-Functions Matter

Think of navigating a gridworld. Being in a particular cell (state) might be moderately good, but:

  • Moving right might lead you toward the goal (high Q-value)
  • Moving left might lead you into a wall (low Q-value)

The Q-function captures these distinctions. Once you know Q(s, a) for all actions in a state, choosing the optimal action becomes straightforward: pick the action with the highest Q-value!

Relationship to State Value

The state value V(s) and Q-function are related. The value of a state is simply the Q-value of the best action you could take from that state (under an optimal policy).

Key Takeaway: Q-functions (Q(s,a)) evaluate state-action pairs, telling you the expected return from taking action a in state s. They're essential for action selection in reinforcement learning because they directly compare different choices.