Course contentsShow
← Machine Learning and Deep Learning
Lesson 2191 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Boltzmann Exploration (Softmax)

Using temperature-controlled probability distributions over actions for smooth exploration-exploitation balance.

Boltzmann Exploration (Softmax)

What you'll learn: How to balance exploration and exploitation using temperature-controlled probability distributions that smoothly favor better actions while still trying alternatives.

The Core Idea

Unlike epsilon-greedy (which randomly picks any action with probability ε) or UCB (which uses confidence intervals), Boltzmann exploration converts action values into selection probabilities using the softmax function. Actions with higher estimated values get higher probabilities, but all actions retain some chance of being chosen.

The key is the temperature parameter τ (tau):

P(a) = exp(Q(a)/τ) / Σ exp(Q(a')/τ)

Where:

  • Q(a) is the estimated value of action a
  • τ controls how "greedy" the selection is
  • The denominator normalizes to create valid probabilities

How Temperature Works

Think of temperature like decision confidence:

  • High τ (hot): All actions get nearly equal probability → more exploration
  • Low τ (cold): Best actions dominate the probability → more exploitation
  • τ → 0: Approaches pure greedy selection
  • τ → ∞: Approaches uniform random selection

This gives you smooth control over the exploration-exploitation balance, unlike the hard cutoff of epsilon-greedy.

When to Use It

Boltzmann exploration shines when:

  • Action values have meaningful relative differences
  • You want proportional exploration (trying good alternatives more than bad ones)
  • You can tune or anneal the temperature over time

It's especially popular in settings where the gap between good and mediocre actions matters, not just "best vs everything else."

Key Takeaway: Boltzmann exploration uses temperature-controlled softmax probabilities to create a smooth, graded exploration strategy where better actions are preferred but all remain possible, providing fine-grained control over exploration-exploitation balance.