Course contentsShow
← Machine Learning and Deep Learning
Lesson 2267 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

The REINFORCE Algorithm Structure

Complete algorithm flow: sample episodes, compute returns, calculate gradients, and update policy parameters.

The REINFORCE Algorithm Structure

What you'll learn: How REINFORCE orchestrates sampling, return calculation, gradient computation, and policy updates into a complete training algorithm.

The Complete Algorithm Flow

REINFORCE is your first complete policy gradient algorithm. Think of it like a feedback loop: try actions, see what works, then nudge your policy toward those successful behaviors.

Here's the step-by-step structure:

1. Sample Episodes

Generate complete episodes by running your current policy π_θ in the environment. Collect the sequence of states, actions, and rewards until termination.

2. Compute Returns

For each timestep t in the episode, calculate the return G_t (sum of discounted future rewards from that point onward). This tells you how good being in that state and taking that action turned out to be.

3. Calculate Gradients

For each timestep, compute the policy gradient:

∇_θ log π_θ(a_t | s_t) × G_t

This gradient points in the direction that increases the probability of actions that led to high returns.

4. Update Policy Parameters

Accumulate gradients across all timesteps in the episode, then update θ using gradient ascent:

θ ← θ + α × ∇_θ J(θ)

Why This Structure Works

REINFORCE is Monte Carlo (waits for complete episodes) and on-policy (learns from its own experience). Each episode provides an unbiased but high-variance estimate of the true gradient. The algorithm gradually shifts probability mass toward actions that empirically led to better outcomes.

The beauty is its simplicity: no value functions, no bootstrapping—just direct policy optimization through trial and feedback.

Key Takeaway: REINFORCE structures policy learning as a cycle: sample full episodes → compute actual returns → calculate policy gradients using the score function → update parameters to reinforce successful actions.