Course contentsShow
← Machine Learning and Deep Learning
Lesson 2150 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

The Bellman Expectation Equation for Q

Express Q(s,a) recursively using immediate rewards and expected Q-values of successor state-action pairs.

The Bellman Expectation Equation for Q

What you'll learn: How to express Q-values recursively using immediate rewards and the expected values of future state-action pairs.

The Core Idea

You've learned that Q(s,a) tells us the expected return from taking action a in state s and following our policy thereafter. The Bellman Expectation Equation for Q gives us a powerful recursive relationship: the value of taking an action now equals the immediate reward plus the discounted value of whatever we expect to do next.

Think of it like planning a road trip. The total value of choosing a particular route from your current location isn't just about the scenery on that immediate stretch—it's that scenery plus the expected value of all the choices you'll make from wherever that route takes you.

The Equation

For a policy π, the Bellman Expectation Equation for Q is:

Q^π(s,a) = R(s,a) + γ Σ P(s'|s,a) Σ π(a'|s') Q^π(s',a')

Breaking this down:

  • R(s,a): The immediate reward for taking action a in state s
  • γ: The discount factor (how much we value future rewards)
  • P(s'|s,a): Probability of transitioning to state s' after taking action a
  • π(a'|s'): Probability our policy chooses action a' in the next state s'
  • Q^π(s',a'): The Q-value of the next state-action pair

The equation says: "To find Q(s,a), take your immediate reward and add the discounted expected Q-value of whatever actions your policy will choose from the successor states."

Key Takeaway: The Bellman Expectation Equation for Q expresses action-values recursively—the Q-value of a state-action pair equals the immediate reward plus the discounted expected Q-value of following the policy from successor states.