Course contentsShow
← Machine Learning and Deep Learning
Lesson 2144 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Optimal Policies and Optimality

Defining optimal policy π* that maximizes value in all states: V*(s) ≥ V^π(s) for all π.

Optimal Policies and Optimality

What you'll learn: You'll understand what makes a policy "optimal" in reinforcement learning and how we formally define the best possible behavior an agent can achieve.

The Gold Standard of Decision Making

In reinforcement learning, we don't just want a policy—we want the best policy. An optimal policy, denoted π*, is one that maximizes the expected return from every state. It's the decision-making strategy that does better than (or at least as well as) every other possible policy.

Formal Definition

A policy π* is optimal if its state value function satisfies:

V*(s) ≥ V^π(s) for all policies π and all states s

In plain English: Starting from any state, following the optimal policy gives you at least as much cumulative reward as following any other policy. The optimal value function V*(s) represents the maximum value achievable from state s.

Similarly, the optimal action-value function Q*(s, a) tells you the best possible value when taking action a from state s and then following the optimal policy thereafter.

Why This Matters

Think of it like finding the absolute best route through a city. While many routes get you to your destination, the optimal policy is the one that maximizes your objectives (shortest time, least traffic, scenic views—whatever you're rewarding). No other strategy can do better.

Importantly, an optimal policy always exists in MDPs, though there may be multiple optimal policies that achieve the same maximum value. Finding π* is the ultimate goal of many RL algorithms.

Key Takeaway: The optimal policy π* is the gold standard that maximizes value in every state, formally defined by V*(s) ≥ V^π(s) for all policies π—it's the best possible behavior an agent can learn.