Policies: Deterministic vs Stochastic
What you'll learn: How to represent an agent's behavior as a policy that maps states to actions, either with certainty or probability.
The Core Idea
A policy is the strategy your agent follows—it tells the agent what action to take in any given state. Think of it as the agent's "rulebook" or "decision-making brain."
Policies come in two flavors:
Deterministic Policies: π(s)
A deterministic policy always chooses the same action for a given state. It's a direct mapping: π(s) = a.
Analogy: Like a vending machine—press button B3, and you always get the same snack. No randomness, no surprises.
When you're in state s, a deterministic policy outputs exactly one action. For example:
- State: "Low health in game" → Action: "Use health potion" (always)
Stochastic Policies: π(a|s)
A stochastic policy defines a probability distribution over actions for each state. It's written as π(a|s), read as "the probability of taking action a given state s."
Analogy: Like choosing your lunch strategy—when hungry (state), you might pick pizza 60% of the time, salad 30%, and skip 10%. There's randomness built in.
For state s, a stochastic policy assigns probabilities to all possible actions, and they sum to 1.0.
Why Stochastic Policies?
Stochastic policies are useful for:
- Exploration: Trying different actions to discover better strategies
- Optimal strategies in some games: Sometimes mixing actions probabilistically is actually optimal (like bluffing in poker)
- Expressiveness: More flexible for learning complex behaviors
Key Takeaway: A policy π represents how your agent behaves—deterministic policies output one action per state π(s), while stochastic policies output a probability distribution over actions π(a|s).