The Bellman Expectation Equation for Q
What you'll learn: How to express Q-values recursively using immediate rewards and the expected values of future state-action pairs.
The Core Idea
You've learned that Q(s,a) tells us the expected return from taking action a in state s and following our policy thereafter. The Bellman Expectation Equation for Q gives us a powerful recursive relationship: the value of taking an action now equals the immediate reward plus the discounted value of whatever we expect to do next.
Think of it like planning a road trip. The total value of choosing a particular route from your current location isn't just about the scenery on that immediate stretch—it's that scenery plus the expected value of all the choices you'll make from wherever that route takes you.
The Equation
For a policy π, the Bellman Expectation Equation for Q is:
Q^π(s,a) = R(s,a) + γ Σ P(s'|s,a) Σ π(a'|s') Q^π(s',a')
Breaking this down:
- R(s,a): The immediate reward for taking action
ain states - γ: The discount factor (how much we value future rewards)
- P(s'|s,a): Probability of transitioning to state
s'after taking actiona - π(a'|s'): Probability our policy chooses action
a'in the next states' - Q^π(s',a'): The Q-value of the next state-action pair
The equation says: "To find Q(s,a), take your immediate reward and add the discounted expected Q-value of whatever actions your policy will choose from the successor states."
Key Takeaway: The Bellman Expectation Equation for Q expresses action-values recursively—the Q-value of a state-action pair equals the immediate reward plus the discounted expected Q-value of following the policy from successor states.