The Bellman Expectation Equation for V
What You'll Learn
You'll learn how to express the value of being in a state as a recursive equation linking immediate rewards to the values of future states.
The Core Idea
The Bellman Expectation Equation is a fundamental recursive relationship that breaks down the value function V(s) into two components:
- The immediate reward you expect from taking actions in state s
- The discounted value of wherever you end up next
Think of it like planning a road trip: the total value of being at your current city equals the enjoyment of activities there plus the (discounted) value of all the cities you might drive to next, weighted by how likely you are to visit each.
The Mathematical Form
For a policy π, the Bellman Expectation Equation states:
V_π(s) = Σ_a π(a|s) · [R(s,a) + γ · Σ_s' P(s'|s,a) · V_π(s')]
Breaking this down:
- π(a|s): probability of taking action a in state s under policy π
- R(s,a): immediate reward for taking action a from state s
- γ (gamma): discount factor for future rewards
- P(s'|s,a): transition probability to next state s'
- V_π(s'): value of the successor state
Why It Matters
This equation is "expectation" because it averages over all possible actions (weighted by your policy) and all possible next states (weighted by transition probabilities). It's recursive because V(s) is defined in terms of other V(s') values, allowing us to solve for all state values simultaneously.
Key Takeaway: The Bellman Expectation Equation expresses that the value of any state equals the expected immediate reward plus the discounted expected value of successor states, creating a self-consistent system we can solve.