Monte Carlo Policy Gradient Derivation
What You'll Learn
You'll see the step-by-step mathematical derivation of the REINFORCE algorithm's gradient estimator using complete episode returns.
The Core Idea
Monte Carlo policy gradient methods estimate the policy gradient by collecting complete episodes and using their total returns. The REINFORCE algorithm is the classic example, and its derivation shows why we can learn from complete trajectories without knowing the environment's dynamics.
Step-by-Step Derivation
We start with our objective: maximize expected return under policy π_θ:
J(θ) = E_τ~π_θ [G(τ)]
where τ represents a trajectory and G(τ) is its return.
Step 1: Gradient of the expectation
∇_θ J(θ) = ∇_θ E_τ [G(τ)]
= E_τ [G(τ) ∇_θ log P(τ|θ)]
This uses the log-derivative trick (score function estimator you learned earlier).
Step 2: Expand trajectory probability
The probability of a trajectory factorizes:
P(τ|θ) = P(s_0) ∏_t π_θ(a_t|s_t) P(s_{t+1}|s_t,a_t)
Step 3: Gradient only depends on policy
When we take ∇_θ log P(τ|θ), environment dynamics cancel out:
∇_θ log P(τ|θ) = ∑_t ∇_θ log π_θ(a_t|s_t)
Step 4: Monte Carlo estimator
Replace expectation with sample average over N episodes:
∇_θ J(θ) ≈ (1/N) ∑_{i=1}^N G(τ^i) ∑_t ∇_θ log π_θ(a_t^i|s_t^i)
This is REINFORCE: each action's gradient is weighted by the total return from that episode.
Key Takeaway: The Monte Carlo policy gradient uses complete episode returns G(τ) to weight policy gradients, requiring no model of environment dynamics—only the log-probability gradients of actions taken and their observed returns.