Temperature Scheduling in Softmax
You'll learn how to gradually shift from exploratory to greedy action selection by adjusting the temperature parameter in Boltzmann (softmax) exploration over time.
The Core Idea
In the previous lesson, you learned about Boltzmann exploration, which uses a temperature parameter T to control how "random" your action selection is. Temperature scheduling is the practice of annealing (gradually reducing) this temperature during training—starting high for broad exploration and ending low for focused exploitation.
Think of it like cooling molten metal: when hot, molecules move freely (exploration); as it cools, they settle into a stable structure (exploitation).
How It Works
Early in training, you set T high (e.g., T = 10.0). The softmax probabilities become nearly uniform, encouraging the agent to try many different actions regardless of their Q-values. As training progresses, you decrease T according to a schedule:
- Linear decay:
T = T_initial - (T_initial - T_final) * (step / max_steps) - Exponential decay:
T = T_initial * decay_rate^step - Step decay: Drop
Tby a factor every N episodes
As T → 0, the softmax distribution becomes sharper, concentrating probability on the highest Q-value action (approaching greedy selection).
Why It Matters
This mirrors decaying epsilon schedules from epsilon-greedy, but offers smoother transitions. Instead of abruptly choosing randomly vs greedily, temperature scheduling provides a continuous spectrum from exploration to exploitation. The agent can still occasionally sample suboptimal actions even late in training, but with decreasing likelihood.
Key Takeaway: Temperature scheduling in softmax exploration gradually reduces the temperature parameter from high to low values during training, smoothly transitioning the agent from exploratory behavior to greedy exploitation of learned Q-values.