Course contentsShow
← Machine Learning and Deep Learning
Lesson 2249 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

From Value Functions to Policies

Why directly optimizing policies can be more effective than learning value functions in certain environments.

From Value Functions to Policies

What you'll learn: Why directly optimizing policies can be more effective than learning value functions in certain environments.

The Fundamental Shift

Up until now, you've focused on value-based methods like Q-Learning and DQN, where you learn Q(s,a) values and derive your policy by choosing actions with maximum Q-value. This indirect approach works well in many scenarios, but it has limitations.

Policy-based methods flip this paradigm: instead of learning values and extracting a policy, you directly learn the policy itself—a mapping from states to actions (or action probabilities).

When Direct Policy Optimization Shines

Think of it like learning to ride a bike. Value-based methods are like calculating the physics of balance at every moment and then deciding what to do. Policy-based methods are like directly learning the muscle memory of riding.

Direct policy optimization excels when:

  • Continuous action spaces: With infinitely many actions (like steering angles), selecting argmax over Q-values becomes intractable
  • Stochastic optimal policies: Some environments require randomness; value functions naturally prefer deterministic policies
  • High-dimensional actions: Computing max over millions of Q-values is expensive
  • Smoother convergence: Small changes to the policy parameters lead to small policy changes, avoiding the instability of switching between discrete actions

The Core Advantage

Value-based methods face a critical bottleneck: they must store and compute values for every state-action pair. Policy methods can represent complex behaviors with relatively few parameters, especially when using function approximation (neural networks).

Key Takeaway: While value functions answer "how good is this action," policy-based methods directly answer "what should I do"—often a more efficient question for continuous, high-dimensional, or inherently stochastic decision problems.