Course contentsShow
← Machine Learning and Deep Learning
Lesson 2263 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

From Value-Based to Policy-Based Methods

Why directly optimizing policies can be more effective than learning value functions in certain domains.

From Value-Based to Policy-Based Methods

What you'll learn: Why directly optimizing policies can be more effective than learning value functions in certain domains.

The Fundamental Shift

In value-based methods like Q-Learning and DQN, we learn how valuable each state-action pair is, then derive a policy by choosing actions with the highest values. This indirect approach works well in many scenarios, but it has inherent limitations.

Policy-based methods flip this paradigm: instead of learning values and extracting a policy, we directly parameterize and optimize the policy itself. We teach the agent what to do, not what's valuable.

Why Direct Policy Optimization Wins

Think of it like learning to drive. A value-based approach is like memorizing the "goodness score" of every possible steering wheel position at every moment, then picking the best one. A policy-based approach is like learning the skill of steering itself—a smooth, direct mapping from what you see to how you turn.

Key advantages:

  1. Continuous action spaces: When actions are continuous (like steering angles or motor torques), value-based methods struggle. You can't maximize over infinitely many Q-values efficiently. Policies can output continuous actions naturally.

  2. Stochastic policies: Sometimes randomness is optimal (rock-paper-scissors, poker). Value-based methods need awkward workarounds like epsilon-greedy. Policy methods learn stochastic behavior naturally.

  3. High-dimensional actions: As action spaces grow, finding argmax Q(s,a) becomes computationally expensive. A policy network just outputs an action directly.

  4. Simpler in partially observable environments: When the state is uncertain, a stochastic policy often converges better than trying to maintain accurate value estimates.

Key Takeaway: Policy-based methods shine when you need continuous actions, inherent stochasticity, or want to avoid the computational burden of action-value maximization—they learn the "how to act" directly rather than the "how good is this" indirectly.