Course contentsShow
← Machine Learning and Deep Learning
Lesson 2192 of 3,53847. Reinforcement Learning: Temporal Difference MethodsFree lesson

Temperature Scheduling in Softmax

Annealing temperature parameters from high (exploration) to low (exploitation) during training.

Temperature Scheduling in Softmax

You'll learn how to gradually shift from exploratory to greedy action selection by adjusting the temperature parameter in Boltzmann (softmax) exploration over time.

The Core Idea

In the previous lesson, you learned about Boltzmann exploration, which uses a temperature parameter T to control how "random" your action selection is. Temperature scheduling is the practice of annealing (gradually reducing) this temperature during training—starting high for broad exploration and ending low for focused exploitation.

Think of it like cooling molten metal: when hot, molecules move freely (exploration); as it cools, they settle into a stable structure (exploitation).

How It Works

Early in training, you set T high (e.g., T = 10.0). The softmax probabilities become nearly uniform, encouraging the agent to try many different actions regardless of their Q-values. As training progresses, you decrease T according to a schedule:

  • Linear decay: T = T_initial - (T_initial - T_final) * (step / max_steps)
  • Exponential decay: T = T_initial * decay_rate^step
  • Step decay: Drop T by a factor every N episodes

As T → 0, the softmax distribution becomes sharper, concentrating probability on the highest Q-value action (approaching greedy selection).

Why It Matters

This mirrors decaying epsilon schedules from epsilon-greedy, but offers smoother transitions. Instead of abruptly choosing randomly vs greedily, temperature scheduling provides a continuous spectrum from exploration to exploitation. The agent can still occasionally sample suboptimal actions even late in training, but with decreasing likelihood.

Key Takeaway: Temperature scheduling in softmax exploration gradually reduces the temperature parameter from high to low values during training, smoothly transitioning the agent from exploratory behavior to greedy exploitation of learned Q-values.