Course contentsShow
← Machine Learning and Deep Learning
Lesson 2213 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Epsilon-Greedy Exploration in DQN

Balancing exploration and exploitation with decaying epsilon schedules during training.

Epsilon-Greedy Exploration in DQN

You'll learn how to balance exploration and exploitation in Deep Q-Networks using decaying epsilon schedules to improve learning efficiency.

The Core Challenge

In DQN, your agent uses a neural network to estimate Q-values for different actions. But how does it decide which action to take? Simply choosing the action with the highest Q-value (greedy exploitation) would cause the agent to get stuck repeating early patterns without discovering better strategies. You need controlled randomness.

Epsilon-Greedy: The Solution

Epsilon-greedy exploration works exactly as you learned in Q-Learning and bandits, but now it's applied to your neural network's predictions:

  • With probability ε (epsilon): choose a random action (explore)
  • With probability 1-ε: choose the action with highest Q-value from the network (exploit)

The Decaying Schedule

Here's the key difference in DQN: epsilon should decay over time. Think of it like learning to ride a bike—you need training wheels (exploration) at first, but gradually remove them as you gain confidence (exploitation).

Common decay strategies:

  • Linear decay: ε decreases steadily from 1.0 → 0.1 over the first million steps
  • Exponential decay: ε = ε × decay_rate after each episode
  • Step decay: ε drops at specific milestones

Early training (ε ≈ 1.0): Agent explores randomly, gathering diverse experiences for the replay buffer.

Late training (ε ≈ 0.1): Agent mostly exploits learned Q-values, with occasional exploration to avoid local optima.

You typically never decay epsilon all the way to zero—keeping a small residual (0.01-0.1) ensures continued exploration even after convergence.

Key Takeaway: Epsilon-greedy exploration with a decaying schedule allows DQN to gradually shift from random exploration to confident exploitation, ensuring the agent discovers good strategies early while refining them later.