Value Functions: State Value V
What you'll learn: How to quantify the long-term "goodness" of being in a particular state under a given policy.
The Core Idea
In reinforcement learning, knowing the immediate reward isn't enough—you need to understand how valuable a state is in the long run. The state value function V^π(s) tells you exactly this: it's the expected total return (cumulative discounted reward) you'll receive starting from state s and following policy π thereafter.
Mathematically:
V^π(s) = E[G_t | S_t = s]
Where G_t is the return (the sum of discounted future rewards we covered in the previous lesson), and the expectation E[...] accounts for all the randomness: stochastic policies, probabilistic state transitions, and varying future rewards.
Understanding Through Analogy
Think of V^π(s) like evaluating a neighborhood's long-term value. Standing at a particular intersection (state s), if you follow your usual driving habits (policy π), what's your expected quality of life going forward? A neighborhood near parks, schools, and work might have high V^π—not because of immediate reward, but because following your policy from there leads to good outcomes over time.
Why It Matters
The state value function is fundamental because:
- It summarizes the quality of states under a specific policy
- It helps compare different states to guide decision-making
- It's the foundation for policy evaluation and improvement algorithms
- Unlike raw returns (which vary per episode), V^π provides a stable expected measure
Key Takeaway: The state value function V^π(s) quantifies the expected cumulative discounted reward from state s under policy π, capturing the long-term value of being in that state rather than just immediate rewards.