Course contentsShow
← Machine Learning and Deep Learning
Lesson 2188 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Decaying Epsilon Schedules

Gradually reducing exploration over time to shift from learning to exploitation as estimates improve.

Decaying Epsilon Schedules

What You'll Learn

You'll learn how to gradually reduce exploration over time so your agent shifts from learning new things to exploiting what it already knows.

The Core Idea

In epsilon-greedy exploration, you choose random actions with probability epsilon (ε) and greedy actions with probability (1-ε). But using a fixed epsilon throughout training isn't optimal. Early on, your Q-value estimates are poor, so heavy exploration helps. Later, as estimates improve, you want to exploit what you've learned.

A decaying epsilon schedule solves this by starting with high ε (lots of exploration) and gradually reducing it toward zero (mostly exploitation). Think of it like learning to ride a bike: at first, you experiment wildly with balance and pedaling. As you improve, you rely more on what works and experiment less.

Common Decay Strategies

Linear Decay

Reduce ε by a fixed amount each episode:

ε = max(ε_min, ε_start - decay_rate × episode)

Exponential Decay

Multiply ε by a factor less than 1:

ε = max(ε_min, ε_start × decay_rate^episode)

Step Decay

Drop ε at specific milestones (e.g., every 100 episodes).

Why It Matters

Without decay, you either explore too much (never converging to optimal behavior) or exploit too early (getting stuck in suboptimal policies). The schedule balances learning breadth early with performance optimization later.

Key Takeaway: Decaying epsilon schedules systematically reduce exploration as training progresses, allowing agents to transition from discovery to performance optimization as their value estimates become reliable.