Temperature Parameter in Contrastive Loss
What You'll Learn
You'll understand how the temperature hyperparameter τ (tau) controls how sharply the model distinguishes between positive and negative pairs in contrastive learning.
The Core Idea
The temperature parameter is a scalar that divides the similarity scores before applying the softmax in the InfoNCE loss. It directly controls the "sharpness" of the probability distribution over negative examples.
Think of temperature like adjusting the contrast on a photo: a low temperature (e.g., 0.1) creates high contrast—making the model very confident and forcing it to strongly separate positives from negatives. A high temperature (e.g., 1.0) creates lower contrast—softening the distinctions and making the loss less sensitive to individual hard negatives.
How It Works
In InfoNCE loss, similarities are computed as dot products and then scaled by temperature:
exp(sim(z_i, z_j) / τ)
- Lower τ (0.05–0.1): Exponentiates differences more aggressively. Even small similarity differences become large probability differences. The model must work harder to distinguish positives.
- Higher τ (0.5–1.0): Smooths out the distribution. All negatives receive more uniform attention, reducing the penalty for mistakes.
Why It Matters
Temperature is crucial for training stability and representation quality. Too low, and gradients become unstable (the model is overconfident). Too high, and the loss provides weak learning signals (everything looks similar). Typical values range from 0.07 to 0.5, with SimCLR using 0.5 as default.
Key Takeaway: The temperature parameter controls how concentrated the probability distribution is over negatives—lower temperatures create sharper, more discriminative representations but require careful tuning to avoid training instability.