Course contentsShow
← Machine Learning and Deep Learning
Lesson 2147 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

The Value Function: State Values in MDPs

Define state-value functions V(s) that measure the expected return from a state under a given policy.

The Value Function: State Values in MDPs

What you'll learn: How to measure the "goodness" of being in a particular state by calculating expected future rewards.

The Core Idea

In reinforcement learning, not all states are created equal. Some positions put you closer to success than others. The state-value function V(s) answers the question: "If I start in state s and follow a specific policy from here on, what's the expected total return I'll get?"

Think of it like evaluating different locations in a city when you're trying to get to work. Being at a subway station two stops away is more valuable than being five stops away—not because of immediate reward, but because of where it positions you for the future.

Mathematical Definition

The state-value function under policy π is defined as:

V^π(s) = E_π[G_t | S_t = s]

Where G_t is the return (cumulative discounted reward) starting from time t. In other words, V^π(s) is the expected return when starting in state s and following policy π thereafter.

Why It Matters

The value function encodes accumulated wisdom about states. It looks beyond immediate rewards to capture long-term consequences. A state might have zero immediate reward but high value because it leads to rewarding future states. Conversely, a state with small immediate reward might have low value if it traps you far from your goal.

The value function depends on the policy—different strategies for acting produce different expected returns from the same state.

Key Takeaway: The state-value function V^π(s) quantifies how good it is to be in a state under a given policy by computing the expected cumulative discounted reward from that state forward.