Course contentsShow
← Machine Learning and Deep Learning
Lesson 2268 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Return Calculation in REINFORCE

Computing discounted cumulative rewards from each timestep to episode end for gradient weighting.

Return Calculation in REINFORCE

What You'll Learn

You'll learn how to compute the discounted cumulative rewards (returns) from each timestep to episode end, which weight the policy gradient updates in REINFORCE.

The Core Idea

In REINFORCE, we need to answer: "How good was taking action a at time t?" The answer is the return — the total discounted reward we received from that timestep forward until the episode ended.

For each timestep t in an episode, the return G_t is calculated as:

G_t = r_t + γ·r_(t+1) + γ²·r_(t+2) + ... + γ^(T-t)·r_T

Where:

  • r_t is the immediate reward at time t
  • γ (gamma) is the discount factor (0 < γ ≤ 1)
  • T is the final timestep of the episode

Why This Matters

The return tells us the total consequence of an action, not just its immediate reward. A sacrifice now (low reward) might lead to victory later (high total return). These returns become the weights in our gradient: actions leading to high returns get reinforced; those leading to low returns get discouraged.

Efficient Computation

We compute returns backward from episode end:

G_T = r_T
G_(T-1) = r_(T-1) + γ·G_T
G_(T-2) = r_(T-2) + γ·G_(T-1)
...

This backward pass ensures each return captures all future rewards efficiently.

Key Takeaway: Returns in REINFORCE are discounted sums of future rewards computed backward from episode end, providing the scalar "credit assignment" signal that weights how strongly we update the policy at each timestep.