Policy Parameterization with Neural Networks
What You'll Learn
You'll learn how to use neural networks to represent stochastic policies that output probability distributions over actions, enabling policy-based reinforcement learning in complex environments.
The Core Idea
In policy-based methods, we need a way to map states to actions. Instead of learning Q-values and deriving a policy from them, we directly learn a parameterized policy π_θ(a|s), where θ represents the neural network's weights.
The neural network takes a state as input and outputs a probability distribution over all possible actions. For discrete actions, the output layer typically uses a softmax activation, producing probabilities that sum to 1. For continuous actions, the network often outputs parameters of a Gaussian distribution (mean and standard deviation).
How It Works
Discrete Actions:
- Input: state representation
- Hidden layers: feature extraction (like any neural network)
- Output layer: softmax over action space
- Result: π_θ(a|s) gives probability for each action
Continuous Actions:
- Input: state representation
- Hidden layers: feature extraction
- Output layer: mean μ(s) and log standard deviation log σ(s)
- Result: sample actions from Normal(μ(s), σ(s))
Analogy
Think of the neural network as a decision advisor that doesn't give you one answer, but instead tells you the probability of each option being good. In a game, it might say "move left: 70%, move right: 25%, jump: 5%"—then you sample from this distribution to choose your action.
The beauty of this approach is that the network's parameters are differentiable, so we can use gradient ascent to improve the policy directly based on the returns we observe.
Key Takeaway: Neural networks naturally represent stochastic policies by outputting probability distributions over actions (via softmax for discrete or Gaussian parameters for continuous), enabling gradient-based policy optimization in complex state spaces.