Course contentsShow
← Machine Learning and Deep Learning
Lesson 2266 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Monte Carlo Policy Gradient Derivation

Step-by-step derivation of the REINFORCE gradient estimator using complete episode returns.

Monte Carlo Policy Gradient Derivation

What You'll Learn

You'll see the step-by-step mathematical derivation of the REINFORCE algorithm's gradient estimator using complete episode returns.

The Core Idea

Monte Carlo policy gradient methods estimate the policy gradient by collecting complete episodes and using their total returns. The REINFORCE algorithm is the classic example, and its derivation shows why we can learn from complete trajectories without knowing the environment's dynamics.

Step-by-Step Derivation

We start with our objective: maximize expected return under policy π_θ:

J(θ) = E_τ~π_θ [G(τ)]

where τ represents a trajectory and G(τ) is its return.

Step 1: Gradient of the expectation

∇_θ J(θ) = ∇_θ E_τ [G(τ)]
          = E_τ [G(τ) ∇_θ log P(τ|θ)]

This uses the log-derivative trick (score function estimator you learned earlier).

Step 2: Expand trajectory probability

The probability of a trajectory factorizes:

P(τ|θ) = P(s_0) ∏_t π_θ(a_t|s_t) P(s_{t+1}|s_t,a_t)

Step 3: Gradient only depends on policy

When we take ∇_θ log P(τ|θ), environment dynamics cancel out:

∇_θ log P(τ|θ) = ∑_t ∇_θ log π_θ(a_t|s_t)

Step 4: Monte Carlo estimator

Replace expectation with sample average over N episodes:

∇_θ J(θ) ≈ (1/N) ∑_{i=1}^N G(τ^i) ∑_t ∇_θ log π_θ(a_t^i|s_t^i)

This is REINFORCE: each action's gradient is weighted by the total return from that episode.

Key Takeaway: The Monte Carlo policy gradient uses complete episode returns G(τ) to weight policy gradients, requiring no model of environment dynamics—only the log-probability gradients of actions taken and their observed returns.