Return and Cumulative Reward
What you'll learn: How to calculate the total discounted reward an agent expects to receive from any point in time.
The Core Idea
In reinforcement learning, an agent doesn't just care about the immediate reward—it needs to think ahead. The return (often denoted G_t) is the total reward an agent will accumulate from timestep t onward, but with a twist: future rewards are discounted to reflect that immediate rewards are more valuable than distant ones.
The mathematical formula is:
G_t = R_{t+1} + γ R_{t+2} + γ² R_{t+3} + γ³ R_{t+4} + ...
Or more compactly:
G_t = Σ(k=0 to ∞) γ^k R_{t+k+1}
Where:
G_tis the return starting from timetR_{t+k+1}is the reward receivedksteps into the futureγ(gamma) is the discount factor (previously learned)
Why Discounting Matters
Think of it like money: 100 a year from now. Similarly, a reward now is more certain and useful than the same reward 100 steps later. The discount factor γ (between 0 and 1) mathematically encodes this preference.
- If
γ = 0.9, a reward 10 steps away is worth only0.9^10 ≈ 0.35times its face value - If
γ = 0, the agent is completely myopic (only cares about immediate reward) - If
γ = 1, all future rewards matter equally (only valid for episodic tasks)
Computing Return in Practice
The return tells us "how good is it to be in this state and follow this policy?" It's the foundation for value functions and policy evaluation.
Key Takeaway: The return G_t is the discounted sum of all future rewards from timestep t, balancing immediate gains against long-term planning through the discount factor γ.