Sentence Transformers Architecture
You'll learn: How sentence-BERT models produce fixed-size semantic embeddings optimized for measuring text similarity.
The Core Innovation
Traditional BERT models weren't designed for efficient similarity comparison. To compare two sentences, you'd need to feed them together through BERT and train a classifier—impractical when searching millions of documents. Sentence Transformers solve this by generating a single, fixed-size vector (embedding) for each sentence independently, making similarity a simple vector comparison.
Architecture Overview
Sentence Transformers build on pre-trained models like BERT, RoBERTa, or DistilBERT, adding a pooling layer on top:
1. Base Transformer Encoder
The foundation remains a standard transformer (like BERT) that processes tokens through multi-head attention and feed-forward layers.
2. Pooling Strategy
The key architectural choice: how to convert variable-length token embeddings into a single fixed-size sentence vector:
- CLS token pooling: Use the
[CLS]token's final hidden state - Mean pooling: Average all token embeddings (most common)
- Max pooling: Take element-wise maximum across tokens
3. Training Objective
Unlike BERT's masked language modeling, sentence transformers use contrastive learning:
- Siamese/triplet networks: Train with (anchor, positive, negative) sentence triplets
- Cosine similarity loss: Ensure similar sentences have high cosine similarity
- Softmax loss on pairs: Classify whether sentence pairs are similar
Why This Matters for RAG
The architecture produces embeddings where semantic similarity = vector proximity. You encode a document once, store its embedding, then find relevant context through fast vector operations—no need to re-encode pairs.
Key Takeaway: Sentence Transformers add pooling and contrastive training to base transformers, creating bi-encoders that independently embed text into semantically meaningful fixed-size vectors perfect for efficient retrieval.