Course contentsShow
← Machine Learning and Deep Learning
Lesson 2552 of 3,53855. Self-Supervised LearningFree lesson

Temperature Parameter in Contrastive Loss

Learn how the temperature hyperparameter controls the concentration of the distribution over negative examples.

Temperature Parameter in Contrastive Loss

What You'll Learn

You'll understand how the temperature hyperparameter τ (tau) controls how sharply the model distinguishes between positive and negative pairs in contrastive learning.

The Core Idea

The temperature parameter is a scalar that divides the similarity scores before applying the softmax in the InfoNCE loss. It directly controls the "sharpness" of the probability distribution over negative examples.

Think of temperature like adjusting the contrast on a photo: a low temperature (e.g., 0.1) creates high contrast—making the model very confident and forcing it to strongly separate positives from negatives. A high temperature (e.g., 1.0) creates lower contrast—softening the distinctions and making the loss less sensitive to individual hard negatives.

How It Works

In InfoNCE loss, similarities are computed as dot products and then scaled by temperature:

exp(sim(z_i, z_j) / τ)
  • Lower τ (0.05–0.1): Exponentiates differences more aggressively. Even small similarity differences become large probability differences. The model must work harder to distinguish positives.
  • Higher τ (0.5–1.0): Smooths out the distribution. All negatives receive more uniform attention, reducing the penalty for mistakes.

Why It Matters

Temperature is crucial for training stability and representation quality. Too low, and gradients become unstable (the model is overconfident). Too high, and the loss provides weak learning signals (everything looks similar). Typical values range from 0.07 to 0.5, with SimCLR using 0.5 as default.

Key Takeaway: The temperature parameter controls how concentrated the probability distribution is over negatives—lower temperatures create sharper, more discriminative representations but require careful tuning to avoid training instability.