Stochastic vs Deterministic Policies
You'll learn when to output probabilities over actions versus selecting a single definite action.
The Core Distinction
In policy-based reinforcement learning, your parameterized policy can make decisions in two fundamentally different ways:
A deterministic policy directly maps states to specific actions: given state s, always take action a. It's like a GPS giving you one clear direction: "Turn left."
A stochastic policy outputs a probability distribution over actions: given state s, sample from a distribution over possible actions. It's like asking a local for directions and getting: "Turn left 70% of the time, or try the shortcut 30% of the time."
When to Use Each
Deterministic policies work well when:
- Your environment is deterministic and fully observable
- You want efficient exploitation after learning
- Continuous action spaces with clear optimal actions (e.g., robotic control)
Stochastic policies are essential when:
- You need built-in exploration during training
- The optimal strategy genuinely requires randomness (rock-paper-scissors, poker)
- Dealing with partial observability or aliased states
- You want the policy gradient theorem to apply cleanly (stochastic policies provide natural gradients)
The Policy-Based Advantage
Stochastic policies are particularly powerful in policy gradient methods because they enable gradient-based learning through probability distributions. Instead of learning which action is best, you learn how to adjust the probabilities of each action.
Key Takeaway: Deterministic policies pick one action per state; stochastic policies sample from action probabilities. Use stochastic policies for exploration and policy gradients, deterministic for efficient execution in well-understood environments.