From Value Functions to Policies
What you'll learn: Why directly optimizing policies can be more effective than learning value functions in certain environments.
The Fundamental Shift
Up until now, you've focused on value-based methods like Q-Learning and DQN, where you learn Q(s,a) values and derive your policy by choosing actions with maximum Q-value. This indirect approach works well in many scenarios, but it has limitations.
Policy-based methods flip this paradigm: instead of learning values and extracting a policy, you directly learn the policy itself—a mapping from states to actions (or action probabilities).
When Direct Policy Optimization Shines
Think of it like learning to ride a bike. Value-based methods are like calculating the physics of balance at every moment and then deciding what to do. Policy-based methods are like directly learning the muscle memory of riding.
Direct policy optimization excels when:
- Continuous action spaces: With infinitely many actions (like steering angles), selecting argmax over Q-values becomes intractable
- Stochastic optimal policies: Some environments require randomness; value functions naturally prefer deterministic policies
- High-dimensional actions: Computing max over millions of Q-values is expensive
- Smoother convergence: Small changes to the policy parameters lead to small policy changes, avoiding the instability of switching between discrete actions
The Core Advantage
Value-based methods face a critical bottleneck: they must store and compute values for every state-action pair. Policy methods can represent complex behaviors with relatively few parameters, especially when using function approximation (neural networks).
Key Takeaway: While value functions answer "how good is this action," policy-based methods directly answer "what should I do"—often a more efficient question for continuous, high-dimensional, or inherently stochastic decision problems.