The Actor: Parameterized Policy Networks
What you'll learn: How the actor network transforms states into action probabilities or continuous values using learnable parameters.
What Is the Actor?
In actor-critic methods, the actor is a neural network that directly represents your policy—it's the component that decides what action to take given a state. Unlike value-based methods (like DQN) that learn Q-values and derive actions indirectly, the actor explicitly outputs actions or action distributions.
How the Actor Network Works
The actor takes the current state as input and produces outputs that depend on your action space:
For Discrete Actions
The network outputs a probability distribution over all possible actions using a softmax activation on the final layer. Each output neuron corresponds to one action, and the softmax ensures probabilities sum to 1.
Analogy: Think of it like a weather forecaster giving probabilities: 60% chance of rain (action 1), 30% sunny (action 2), 10% snow (action 3).
For Continuous Actions
The network outputs parameters of a continuous distribution—typically the mean (and sometimes variance) of a Gaussian distribution. You then sample from this distribution to get the actual action value.
Analogy: Like a thermostat controller that outputs "set temperature to 72°F ± 2°" rather than choosing from preset options.
Why Parameterization Matters
By using neural network weights as learnable parameters (θ), we can represent complex policies that map high-dimensional states to appropriate actions. The policy gradient theorem tells us how to update these parameters to improve expected return.
Key Takeaway: The actor network parameterizes your policy π(a|s; θ), outputting action probabilities for discrete spaces or distribution parameters for continuous spaces, making the policy differentiable and optimizable through gradient ascent.