Contrastive Loss for Siamese Networks
You'll learn how Siamese networks use contrastive loss to learn embeddings where similar items cluster together and dissimilar items stay far apart.
The Core Idea
In few-shot learning, Siamese networks process pairs of examples through twin networks that share weights. But how do we train them? Contrastive loss is the key: it's a loss function that explicitly teaches the network to minimize distance between similar pairs (positive pairs) and maximize distance between dissimilar pairs (negative pairs) in the embedding space.
Think of it like organizing a library: books on the same topic should be shelved close together, while books on different topics should be separated. Contrastive loss is the "librarian" that enforces this organization rule.
The Mathematical Formulation
For a pair of examples (x₁, x₂) with embeddings f(x₁) and f(x₂):
L_contrastive = (1 - y) * ½ * D² + y * ½ * max(0, margin - D)²
Where:
- D = distance between embeddings (often Euclidean:
||f(x₁) - f(x₂)||) - y = label (0 if similar pair, 1 if dissimilar)
- margin = minimum distance for dissimilar pairs
For similar pairs (y=0): loss is just D², pulling embeddings together.
For dissimilar pairs (y=1): loss is zero if D ≥ margin, otherwise penalizes pairs that are too close.
How It Works in Practice
The loss creates opposing forces: positive pairs experience an attractive force (reducing distance), while negative pairs experience a repulsive force (increasing distance up to the margin). This shapes the embedding space to be discriminative for downstream tasks like nearest-neighbor classification.
Key Takeaway: Contrastive loss trains Siamese networks by pulling similar examples together and pushing dissimilar examples apart by at least a margin, creating a structured embedding space ideal for few-shot recognition.