Boltzmann Exploration (Softmax)
What you'll learn: How to balance exploration and exploitation using temperature-controlled probability distributions that smoothly favor better actions while still trying alternatives.
The Core Idea
Unlike epsilon-greedy (which randomly picks any action with probability ε) or UCB (which uses confidence intervals), Boltzmann exploration converts action values into selection probabilities using the softmax function. Actions with higher estimated values get higher probabilities, but all actions retain some chance of being chosen.
The key is the temperature parameter τ (tau):
P(a) = exp(Q(a)/τ) / Σ exp(Q(a')/τ)
Where:
Q(a)is the estimated value of actiona- τ controls how "greedy" the selection is
- The denominator normalizes to create valid probabilities
How Temperature Works
Think of temperature like decision confidence:
- High τ (hot): All actions get nearly equal probability → more exploration
- Low τ (cold): Best actions dominate the probability → more exploitation
- τ → 0: Approaches pure greedy selection
- τ → ∞: Approaches uniform random selection
This gives you smooth control over the exploration-exploitation balance, unlike the hard cutoff of epsilon-greedy.
When to Use It
Boltzmann exploration shines when:
- Action values have meaningful relative differences
- You want proportional exploration (trying good alternatives more than bad ones)
- You can tune or anneal the temperature over time
It's especially popular in settings where the gap between good and mediocre actions matters, not just "best vs everything else."
Key Takeaway: Boltzmann exploration uses temperature-controlled softmax probabilities to create a smooth, graded exploration strategy where better actions are preferred but all remain possible, providing fine-grained control over exploration-exploitation balance.