Discount Factor Gamma
You'll learn why we value immediate rewards more than distant ones using a single parameter γ (gamma).
The Core Idea
In reinforcement learning, an agent receives rewards over time. But should a reward you get right now count the same as a reward you might get 100 steps from now? The discount factor γ (gamma) is a number between 0 and 1 that controls this trade-off.
When calculating the total expected return, future rewards are multiplied by γ raised to the power of how many steps away they are:
- Immediate reward: reward × γ⁰ = reward × 1
- Next step reward: reward × γ¹ = reward × γ
- Two steps away: reward × γ²
- Three steps away: reward × γ³
Why Discount?
Think of γ like interest rates in reverse. A bird in the hand is worth two in the bush! There are practical reasons to discount:
- Uncertainty: Future outcomes are less certain than immediate ones
- Temporal preference: Solving problems sooner is often better
- Mathematical stability: Prevents infinite sums in continuing tasks
- Planning horizon control: Low γ → shortsighted agent; High γ → far-sighted agent
The Spectrum
- γ = 0: Agent is myopic, only cares about immediate reward (very shortsighted)
- γ = 0.9: Balances near-term and medium-term rewards
- γ = 0.99: Values long-term outcomes heavily (patient planner)
- γ = 1: No discounting; all future rewards count equally (can cause issues)
Key Takeaway: The discount factor γ ∈ [0,1] determines how much an agent values future rewards relative to immediate ones, with lower values creating shortsighted behavior and higher values encouraging long-term planning.