Return Calculation in REINFORCE
What You'll Learn
You'll learn how to compute the discounted cumulative rewards (returns) from each timestep to episode end, which weight the policy gradient updates in REINFORCE.
The Core Idea
In REINFORCE, we need to answer: "How good was taking action a at time t?" The answer is the return — the total discounted reward we received from that timestep forward until the episode ended.
For each timestep t in an episode, the return G_t is calculated as:
G_t = r_t + γ·r_(t+1) + γ²·r_(t+2) + ... + γ^(T-t)·r_T
Where:
r_tis the immediate reward at time tγ(gamma) is the discount factor (0 < γ ≤ 1)Tis the final timestep of the episode
Why This Matters
The return tells us the total consequence of an action, not just its immediate reward. A sacrifice now (low reward) might lead to victory later (high total return). These returns become the weights in our gradient: actions leading to high returns get reinforced; those leading to low returns get discouraged.
Efficient Computation
We compute returns backward from episode end:
G_T = r_T
G_(T-1) = r_(T-1) + γ·G_T
G_(T-2) = r_(T-2) + γ·G_(T-1)
...
This backward pass ensures each return captures all future rewards efficiently.
Key Takeaway: Returns in REINFORCE are discounted sums of future rewards computed backward from episode end, providing the scalar "credit assignment" signal that weights how strongly we update the policy at each timestep.