Course contentsShow
← Machine Learning and Deep Learning
Lesson 2138 of 3,53846. Reinforcement Learning: FundamentalsFree lesson

Discount Factor Gamma

Why we discount future rewards: trading off immediate vs long-term value with γ ∈ [0,1].

Discount Factor Gamma

You'll learn why we value immediate rewards more than distant ones using a single parameter γ (gamma).

The Core Idea

In reinforcement learning, an agent receives rewards over time. But should a reward you get right now count the same as a reward you might get 100 steps from now? The discount factor γ (gamma) is a number between 0 and 1 that controls this trade-off.

When calculating the total expected return, future rewards are multiplied by γ raised to the power of how many steps away they are:

  • Immediate reward: reward × γ⁰ = reward × 1
  • Next step reward: reward × γ¹ = reward × γ
  • Two steps away: reward × γ²
  • Three steps away: reward × γ³

Why Discount?

Think of γ like interest rates in reverse. A bird in the hand is worth two in the bush! There are practical reasons to discount:

  1. Uncertainty: Future outcomes are less certain than immediate ones
  2. Temporal preference: Solving problems sooner is often better
  3. Mathematical stability: Prevents infinite sums in continuing tasks
  4. Planning horizon control: Low γ → shortsighted agent; High γ → far-sighted agent

The Spectrum

  • γ = 0: Agent is myopic, only cares about immediate reward (very shortsighted)
  • γ = 0.9: Balances near-term and medium-term rewards
  • γ = 0.99: Values long-term outcomes heavily (patient planner)
  • γ = 1: No discounting; all future rewards count equally (can cause issues)

Key Takeaway: The discount factor γ ∈ [0,1] determines how much an agent values future rewards relative to immediate ones, with lower values creating shortsighted behavior and higher values encouraging long-term planning.