Score Function Estimator
What You'll Learn
You'll learn the log-derivative trick, a clever mathematical technique that lets us compute policy gradients without needing to differentiate through the unpredictable environment dynamics.
The Core Problem
In policy-based RL, we want to optimize a parameterized policy by computing gradients of expected rewards. But there's a problem: our rewards come from an environment we can't differentiate through. How do you take derivatives when part of your computation graph is a black box?
The Log-Derivative Trick
The score function estimator (also called the REINFORCE trick or likelihood ratio method) solves this with a mathematical sleight of hand:
For any probability distribution π(a|s; θ), we can use the identity:
∇_θ π(a|s; θ) = π(a|s; θ) · ∇_θ log π(a|s; θ)
This transforms our gradient computation from:
∇_θ E[R] = E[∇_θ π(a|s; θ) · R] # Can't compute directly
Into:
∇_θ E[R] = E[π(a|s; θ) · ∇_θ log π(a|s; θ) · R] # Can compute!
Why This Works
Think of it like this: instead of asking "how does changing my policy parameters change the probability of actions," we ask "given the actions I already took, how should I adjust my parameters to make good actions more likely?"
The log-probability gradient ∇_θ log π(a|s; θ) points in the direction that increases the likelihood of action a. We weight this by the reward R — positive rewards push us toward that action, negative rewards push away.
The beauty: we only need the log-probability and its gradient, not derivatives of the environment!
Key Takeaway: The score function estimator uses the log-derivative trick to compute policy gradients by reweighting log-probability gradients with observed returns, bypassing the need to differentiate through environment dynamics.