Contrastive Learning for Embeddings
What You'll Learn
You'll discover how contrastive loss functions train embedding models to automatically group similar texts close together while pushing dissimilar ones apart in vector space.
The Core Idea
Think of contrastive learning like organizing a library by reading comprehension: books on similar topics should sit near each other, while unrelated books stay far apart. That's exactly what contrastive loss does for text embeddings.
When training embedding models like Sentence Transformers, contrastive learning works with pairs or triplets of examples:
- Positive pairs: Similar texts (e.g., a question and its correct answer, or two paraphrases)
- Negative pairs: Dissimilar texts (e.g., unrelated sentences)
The loss function mathematically rewards the model when:
- Similar texts have embeddings that are close in vector space (small distance)
- Dissimilar texts have embeddings that are far apart (large distance)
This creates a powerful training signal: the model learns to encode semantic meaning into the geometry of the embedding space itself.
Why This Matters for Bi-Encoders
Remember that bi-encoders encode texts independently and compare embeddings via similarity measures. Contrastive learning is the training method that teaches them what "similar" means. Without it, embeddings would be arbitrary—contrastive loss gives them semantic structure.
Common contrastive loss functions include:
- Triplet loss: Anchor, positive, negative
- InfoNCE loss: Used in many modern systems
- Multiple Negatives Ranking Loss: Efficient batch-based training
Key Takeaway: Contrastive learning trains embeddings by pulling similar examples together and pushing dissimilar ones apart, creating semantically meaningful vector spaces perfect for retrieval and comparison tasks.