Course contentsShow
← Machine Learning and Deep Learning
Lesson 2142 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Value Functions: State Value V

Expected return from a state under policy π: V^π(s) = E[G_t | S_t=s].

Value Functions: State Value V

What you'll learn: How to quantify the long-term "goodness" of being in a particular state under a given policy.

The Core Idea

In reinforcement learning, knowing the immediate reward isn't enough—you need to understand how valuable a state is in the long run. The state value function V^π(s) tells you exactly this: it's the expected total return (cumulative discounted reward) you'll receive starting from state s and following policy π thereafter.

Mathematically:

V^π(s) = E[G_t | S_t = s]

Where G_t is the return (the sum of discounted future rewards we covered in the previous lesson), and the expectation E[...] accounts for all the randomness: stochastic policies, probabilistic state transitions, and varying future rewards.

Understanding Through Analogy

Think of V^π(s) like evaluating a neighborhood's long-term value. Standing at a particular intersection (state s), if you follow your usual driving habits (policy π), what's your expected quality of life going forward? A neighborhood near parks, schools, and work might have high V^π—not because of immediate reward, but because following your policy from there leads to good outcomes over time.

Why It Matters

The state value function is fundamental because:

  • It summarizes the quality of states under a specific policy
  • It helps compare different states to guide decision-making
  • It's the foundation for policy evaluation and improvement algorithms
  • Unlike raw returns (which vary per episode), V^π provides a stable expected measure

Key Takeaway: The state value function V^π(s) quantifies the expected cumulative discounted reward from state s under policy π, capturing the long-term value of being in that state rather than just immediate rewards.