The Value Function: State Values in MDPs
What you'll learn: How to measure the "goodness" of being in a particular state by calculating expected future rewards.
The Core Idea
In reinforcement learning, not all states are created equal. Some positions put you closer to success than others. The state-value function V(s) answers the question: "If I start in state s and follow a specific policy from here on, what's the expected total return I'll get?"
Think of it like evaluating different locations in a city when you're trying to get to work. Being at a subway station two stops away is more valuable than being five stops away—not because of immediate reward, but because of where it positions you for the future.
Mathematical Definition
The state-value function under policy π is defined as:
V^π(s) = E_π[G_t | S_t = s]
Where G_t is the return (cumulative discounted reward) starting from time t. In other words, V^π(s) is the expected return when starting in state s and following policy π thereafter.
Why It Matters
The value function encodes accumulated wisdom about states. It looks beyond immediate rewards to capture long-term consequences. A state might have zero immediate reward but high value because it leads to rewarding future states. Conversely, a state with small immediate reward might have low value if it traps you far from your goal.
The value function depends on the policy—different strategies for acting produce different expected returns from the same state.
Key Takeaway: The state-value function V^π(s) quantifies how good it is to be in a state under a given policy by computing the expected cumulative discounted reward from that state forward.