Course contentsShow
← Machine Learning and Deep Learning
Lesson 2149 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

The Bellman Expectation Equation for V

Derive the recursive relationship expressing V(s) in terms of immediate rewards and successor state values.

The Bellman Expectation Equation for V

What You'll Learn

You'll learn how to express the value of being in a state as a recursive equation linking immediate rewards to the values of future states.

The Core Idea

The Bellman Expectation Equation is a fundamental recursive relationship that breaks down the value function V(s) into two components:

  1. The immediate reward you expect from taking actions in state s
  2. The discounted value of wherever you end up next

Think of it like planning a road trip: the total value of being at your current city equals the enjoyment of activities there plus the (discounted) value of all the cities you might drive to next, weighted by how likely you are to visit each.

The Mathematical Form

For a policy π, the Bellman Expectation Equation states:

V_π(s) = Σ_a π(a|s) · [R(s,a) + γ · Σ_s' P(s'|s,a) · V_π(s')]

Breaking this down:

  • π(a|s): probability of taking action a in state s under policy π
  • R(s,a): immediate reward for taking action a from state s
  • γ (gamma): discount factor for future rewards
  • P(s'|s,a): transition probability to next state s'
  • V_π(s'): value of the successor state

Why It Matters

This equation is "expectation" because it averages over all possible actions (weighted by your policy) and all possible next states (weighted by transition probabilities). It's recursive because V(s) is defined in terms of other V(s') values, allowing us to solve for all state values simultaneously.

Key Takeaway: The Bellman Expectation Equation expresses that the value of any state equals the expected immediate reward plus the discounted expected value of successor states, creating a self-consistent system we can solve.