Temperature in Contrastive Loss
What you'll learn: How the temperature parameter in contrastive learning controls the sharpness of the similarity distribution and influences training dynamics.
The Core Idea
In the InfoNCE loss you just learned, there's a small but mighty parameter called temperature (τ, tau) that sits in the denominator of the softmax. This scalar divides all similarity scores before computing probabilities, fundamentally changing how the model learns to distinguish positive from negative pairs.
How Temperature Works
Temperature controls the concentration of the probability distribution:
-
Low temperature (τ → 0): Makes the distribution very sharp and peaked. The model becomes extremely confident, assigning nearly all probability mass to the highest-scoring pair. Think of it like turning up contrast on an image—differences become stark.
-
High temperature (τ → ∞): Makes the distribution soft and uniform. All pairs receive more similar probabilities, smoothing out differences. This is like lowering contrast—everything blends together.
Learning Dynamics Impact
Low temperature (e.g., 0.05-0.1):
- Model focuses intensely on hard negatives (those similar to positives)
- Training can be unstable early on when representations are random
- Pushes for very tight clusters of similar items
High temperature (e.g., 0.5-1.0):
- More forgiving early in training
- Learns from broader set of negatives
- Slower convergence but potentially more stable
The sweet spot (often 0.07-0.2 in vision tasks) balances learning efficiency with stability—sharp enough to create meaningful distinctions, soft enough to learn from multiple negatives simultaneously.
Key Takeaway: Temperature in contrastive loss acts like a "confidence knob"—lower values make the model more decisive about similarities, while higher values encourage gentler, more distributed learning across all negative pairs.