Course contentsShow
← Machine Learning and Deep Learning
Lesson 2264 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Policy Parameterization with Neural Networks

How to represent stochastic policies using neural networks that output action probability distributions.

Policy Parameterization with Neural Networks

What You'll Learn

You'll learn how to use neural networks to represent stochastic policies that output probability distributions over actions, enabling policy-based reinforcement learning in complex environments.

The Core Idea

In policy-based methods, we need a way to map states to actions. Instead of learning Q-values and deriving a policy from them, we directly learn a parameterized policy π_θ(a|s), where θ represents the neural network's weights.

The neural network takes a state as input and outputs a probability distribution over all possible actions. For discrete actions, the output layer typically uses a softmax activation, producing probabilities that sum to 1. For continuous actions, the network often outputs parameters of a Gaussian distribution (mean and standard deviation).

How It Works

Discrete Actions:

  • Input: state representation
  • Hidden layers: feature extraction (like any neural network)
  • Output layer: softmax over action space
  • Result: π_θ(a|s) gives probability for each action

Continuous Actions:

  • Input: state representation
  • Hidden layers: feature extraction
  • Output layer: mean μ(s) and log standard deviation log σ(s)
  • Result: sample actions from Normal(μ(s), σ(s))

Analogy

Think of the neural network as a decision advisor that doesn't give you one answer, but instead tells you the probability of each option being good. In a game, it might say "move left: 70%, move right: 25%, jump: 5%"—then you sample from this distribution to choose your action.

The beauty of this approach is that the network's parameters are differentiable, so we can use gradient ascent to improve the policy directly based on the returns we observe.

Key Takeaway: Neural networks naturally represent stochastic policies by outputting probability distributions over actions (via softmax for discrete or Gaussian parameters for continuous), enabling gradient-based policy optimization in complex state spaces.