Hard Negatives Mining
What you'll learn: How to strengthen your embedding models by strategically selecting challenging negative examples that push the model to make finer distinctions.
The Core Problem
When training embedding models for semantic search using contrastive learning, you need positive pairs (semantically similar) and negative pairs (dissimilar). A naive approach randomly samples negatives: if your query is "best pizza in Chicago," random negatives might be completely unrelated like "quantum physics equations."
But random negatives are too easy. The model quickly learns to separate them without developing nuanced understanding.
What Are Hard Negatives?
Hard negatives are semantically close but not actually relevant examples. They're tricky cases that force your model to learn subtle differences.
For the query "best pizza in Chicago," hard negatives might be:
- "top-rated Italian restaurants in Chicago" (close topic, wrong focus)
- "best pizza in New York" (right food, wrong city)
- "Chicago deep dish recipe" (related but different intent)
These are hard because they share vocabulary and concepts with the positive answer, but shouldn't match.
Why This Matters
Hard negatives mining dramatically improves embedding quality because:
- The model learns to distinguish fine-grained semantic differences
- It prevents the model from relying on superficial keyword overlap
- Training becomes more efficient—each example teaches more
The Mining Process
During training, you periodically:
- Use your current model to encode a pool of candidates
- Find examples with high similarity scores to your query but aren't true matches
- Use these as negatives in your next training batch
Key Takeaway: Hard negatives mining improves semantic embeddings by forcing models to learn from difficult, semantically-close-but-incorrect examples rather than easy random ones, leading to more discriminative representations.