Epsilon-Greedy Exploration in DQN
You'll learn how to balance exploration and exploitation in Deep Q-Networks using decaying epsilon schedules to improve learning efficiency.
The Core Challenge
In DQN, your agent uses a neural network to estimate Q-values for different actions. But how does it decide which action to take? Simply choosing the action with the highest Q-value (greedy exploitation) would cause the agent to get stuck repeating early patterns without discovering better strategies. You need controlled randomness.
Epsilon-Greedy: The Solution
Epsilon-greedy exploration works exactly as you learned in Q-Learning and bandits, but now it's applied to your neural network's predictions:
- With probability ε (epsilon): choose a random action (explore)
- With probability 1-ε: choose the action with highest Q-value from the network (exploit)
The Decaying Schedule
Here's the key difference in DQN: epsilon should decay over time. Think of it like learning to ride a bike—you need training wheels (exploration) at first, but gradually remove them as you gain confidence (exploitation).
Common decay strategies:
- Linear decay: ε decreases steadily from 1.0 → 0.1 over the first million steps
- Exponential decay: ε = ε × decay_rate after each episode
- Step decay: ε drops at specific milestones
Early training (ε ≈ 1.0): Agent explores randomly, gathering diverse experiences for the replay buffer.
Late training (ε ≈ 0.1): Agent mostly exploits learned Q-values, with occasional exploration to avoid local optima.
You typically never decay epsilon all the way to zero—keeping a small residual (0.01-0.1) ensures continued exploration even after convergence.
Key Takeaway: Epsilon-greedy exploration with a decaying schedule allows DQN to gradually shift from random exploration to confident exploitation, ensuring the agent discovers good strategies early while refining them later.