Contrastive Training for Embedding Models
What You'll Learn
You'll understand how contrastive loss trains embedding models to push similar texts together and pull dissimilar texts apart in vector space.
The Core Idea
Contrastive training is the secret sauce behind powerful embedding models like Sentence Transformers. Instead of just learning to predict words or classify documents, these models learn relationships between texts.
Here's how it works: during training, the model sees pairs or triplets of examples:
- Positive pairs: Similar texts (e.g., a question and its answer, or two paraphrases)
- Negative pairs: Dissimilar texts (e.g., unrelated sentences)
The contrastive loss function rewards the model when it places positive pairs close together in vector space and penalizes it when similar texts end up far apart. Simultaneously, it pushes negative pairs away from each other.
The Magnetic Analogy
Think of embeddings as magnets on a 2D surface. Contrastive training adjusts the "magnetic charge" of similar texts so they attract each other, while dissimilar texts repel. Over thousands of training examples, the model learns a vector space where semantic similarity directly translates to geometric proximity.
Why This Matters for Retrieval
When you later perform vector search in a RAG system, you query the database with an embedded question. Because contrastive training organized the space meaningfully, the nearest neighbors are genuinely relevant documents—not just lexically similar ones.
Key Takeaway: Contrastive training creates semantically meaningful vector spaces by explicitly teaching the model to cluster similar texts and separate dissimilar ones, making dense retrieval possible.