Course contentsShow
← Machine Learning and Deep Learning
Lesson 2153 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

The Bellman Optimality Equation for Q*

Express Q*(s,a) using the optimal value of the best action in successor states.

The Bellman Optimality Equation for Q*

What you'll learn: How to express the optimal action-value Q*(s,a) recursively using the best action in successor states.

Core Idea

You've already seen how V* (optimal state value) relates to itself through the Bellman Optimality Equation. Now we do the same for Q*, the optimal action-value function.

The Bellman Optimality Equation for Q* says: the value of taking action a in state s under an optimal policy equals the immediate reward you get, plus the discounted value of the best possible action in whatever state you land in next.

Mathematically:

Q*(s, a) = E[r + γ · max_{a'} Q*(s', a')]

Or more explicitly with transition probabilities:

Q*(s, a) = Σ p(s', r | s, a) · [r + γ · max_{a'} Q*(s', a')]

Breaking It Down

  • *Q(s, a)**: The optimal value of taking action a from state s
  • r: The immediate reward received
  • γ: Discount factor (how much we care about future rewards)
  • s': The next state we transition to
  • *max_{a'} Q(s', a')**: The value of the best action available in state s'

Analogy: You're choosing which road to take at an intersection. Q*(s,a) tells you the total value of choosing road a: the immediate scenery you'll see, plus (looking ahead) the value of making the best choice at the next intersection you reach.

The key difference from Q's Bellman Expectation Equation? Here we maximize over actions in the next state, because we assume optimal behavior going forward.

Key Takeaway: The Bellman Optimality Equation for Q* recursively defines optimal action-values by combining immediate rewards with the maximum achievable value from successor states.