Course contentsShow
← Machine Learning and Deep Learning
Lesson 2254 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Episode-Based Gradient Estimation

Collecting full trajectories and using Monte Carlo returns to estimate policy gradients.

Episode-Based Gradient Estimation

What you'll learn: How to collect complete episode trajectories and use Monte Carlo returns to estimate policy gradients in reinforcement learning.

The Core Idea

In policy gradient methods, we need to estimate how adjusting our policy parameters will affect expected returns. Episode-based gradient estimation takes a straightforward approach: run the agent through complete episodes, observe what happens, and use the actual returns (total rewards) to guide parameter updates.

Think of it like learning from finished games. Instead of updating your strategy mid-match, you play the entire game, see your final score, then reflect on which actions led to that outcome. If you won, reinforce the actions you took; if you lost, discourage them.

How It Works

The process follows these steps:

  1. Generate a complete trajectory: Run your current policy from start to terminal state, collecting states, actions, and rewards
  2. Compute Monte Carlo returns: For each time step, calculate the total reward from that point onward (the actual return G_t)
  3. Estimate the gradient: Use these observed returns to approximate how the policy should change

The key formula builds on the score function estimator you learned previously:

∇J(θ) ≈ (1/N) Σ_episodes Σ_timesteps ∇log π(a_t|s_t) G_t

Where G_t is the actual return from time t onward in that episode. This is pure Monte Carlo: we wait until the episode ends to know the true outcome.

Why Episode-Based?

This approach is unbiased—we're using real outcomes, not estimates of estimates. The tradeoff is high variance (different episodes can vary wildly) and we must wait for complete episodes, which can be slow in long environments.

Key Takeaway: Episode-based gradient estimation collects full trajectories and uses Monte Carlo returns as the true signal for updating policy parameters—unbiased but potentially high variance.