Parameterized Policies
What you'll learn: How to represent policies as neural networks with learnable parameters that directly map states to probability distributions over actions.
The Core Idea
Until now, we've been learning action values (Q-functions) and deriving policies from them. But what if we could directly learn the policy itself? That's what parameterized policies do.
A parameterized policy uses a neural network with parameters θ (theta) to represent the function π_θ(a|s) — the probability of taking action a in state s. Instead of a lookup table or value function, your policy is now a differentiable function approximator with continuous parameters you can optimize using gradient descent.
How It Works
Think of the neural network as a translator: it takes a state as input and outputs a probability distribution over all possible actions. For discrete actions, the output layer typically uses a softmax activation to ensure valid probabilities (they sum to 1). For continuous actions, the network might output parameters of a Gaussian distribution (mean and variance).
The beauty here is that small changes to the network weights (parameters θ) create smooth, gradual changes in the policy's behavior. This is fundamentally different from value-based methods where small Q-value changes can cause abrupt policy switches.
Why This Matters
Parameterized policies enable:
- Direct optimization of what you care about (the policy)
- Stochastic policies that naturally handle exploration
- Continuous action spaces where selecting "max Q" isn't feasible
- Smooth policy updates that improve learning stability
Key Takeaway: Parameterized policies represent the policy as a neural network π_θ(a|s) with learnable parameters θ, enabling direct policy learning through gradient-based optimization rather than learning values first.