Course contentsShow
← Machine Learning and Deep Learning
Lesson 2253 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Score Function Estimator

The log-derivative trick for computing policy gradients without differentiating through the environment.

Score Function Estimator

What You'll Learn

You'll learn the log-derivative trick, a clever mathematical technique that lets us compute policy gradients without needing to differentiate through the unpredictable environment dynamics.

The Core Problem

In policy-based RL, we want to optimize a parameterized policy by computing gradients of expected rewards. But there's a problem: our rewards come from an environment we can't differentiate through. How do you take derivatives when part of your computation graph is a black box?

The Log-Derivative Trick

The score function estimator (also called the REINFORCE trick or likelihood ratio method) solves this with a mathematical sleight of hand:

For any probability distribution π(a|s; θ), we can use the identity:

∇_θ π(a|s; θ) = π(a|s; θ) · ∇_θ log π(a|s; θ)

This transforms our gradient computation from:

∇_θ E[R] = E[∇_θ π(a|s; θ) · R]  # Can't compute directly

Into:

∇_θ E[R] = E[π(a|s; θ) · ∇_θ log π(a|s; θ) · R]  # Can compute!

Why This Works

Think of it like this: instead of asking "how does changing my policy parameters change the probability of actions," we ask "given the actions I already took, how should I adjust my parameters to make good actions more likely?"

The log-probability gradient ∇_θ log π(a|s; θ) points in the direction that increases the likelihood of action a. We weight this by the reward R — positive rewards push us toward that action, negative rewards push away.

The beauty: we only need the log-probability and its gradient, not derivatives of the environment!

Key Takeaway: The score function estimator uses the log-derivative trick to compute policy gradients by reweighting log-probability gradients with observed returns, bypassing the need to differentiate through environment dynamics.