Epsilon-Greedy Exploration
What you'll learn: How to balance exploration and exploitation using a simple probability-based strategy that randomly explores with probability ε.
The Core Idea
Epsilon-greedy is the most widely used exploration strategy in reinforcement learning. The concept is beautifully simple: with probability ε (epsilon), choose a random action to explore; otherwise (with probability 1-ε), choose the greedy action that currently looks best according to your Q-values.
Think of it like trying restaurants: most of the time (say, 90%) you go to your favorite place, but occasionally (10%) you randomly pick a new restaurant to try—you might discover something even better!
How It Works
At each step, you:
- Generate a random number between 0 and 1
- If it's less than ε: pick a random action (explore)
- Otherwise: pick the action with the highest Q-value (exploit)
Typical values: ε = 0.1 (explore 10% of the time) or ε = 0.05 (explore 5%)
Why It Matters
Remember from "Greedy Action Selection and Its Limitations" that pure greedy selection gets stuck in local optima. Epsilon-greedy solves this by guaranteeing continued exploration, ensuring every state-action pair gets visited infinitely often (in theory), which is necessary for Q-learning and SARSA to converge to optimal policies.
The beauty is its simplicity: just one parameter controls the exploration-exploitation trade-off. Too high? You waste time on bad actions. Too low? You might never discover better strategies.
Key Takeaway: Epsilon-greedy exploration randomly explores with probability ε and exploits greedily otherwise, providing a simple yet effective solution to the exploration-exploitation dilemma that works well in practice for TD methods like Q-learning and SARSA.