Course contentsShow
← Machine Learning and Deep Learning
Lesson 1972 of 3,53842. Retrieval-Augmented Generation: FoundationsFree lesson

Sentence Transformers Architecture

Learn the architecture of sentence-BERT models that generate fixed-size embeddings optimized for semantic similarity.

Sentence Transformers Architecture

You'll learn: How sentence-BERT models produce fixed-size semantic embeddings optimized for measuring text similarity.

The Core Innovation

Traditional BERT models weren't designed for efficient similarity comparison. To compare two sentences, you'd need to feed them together through BERT and train a classifier—impractical when searching millions of documents. Sentence Transformers solve this by generating a single, fixed-size vector (embedding) for each sentence independently, making similarity a simple vector comparison.

Architecture Overview

Sentence Transformers build on pre-trained models like BERT, RoBERTa, or DistilBERT, adding a pooling layer on top:

1. Base Transformer Encoder

The foundation remains a standard transformer (like BERT) that processes tokens through multi-head attention and feed-forward layers.

2. Pooling Strategy

The key architectural choice: how to convert variable-length token embeddings into a single fixed-size sentence vector:

  • CLS token pooling: Use the [CLS] token's final hidden state
  • Mean pooling: Average all token embeddings (most common)
  • Max pooling: Take element-wise maximum across tokens

3. Training Objective

Unlike BERT's masked language modeling, sentence transformers use contrastive learning:

  • Siamese/triplet networks: Train with (anchor, positive, negative) sentence triplets
  • Cosine similarity loss: Ensure similar sentences have high cosine similarity
  • Softmax loss on pairs: Classify whether sentence pairs are similar

Why This Matters for RAG

The architecture produces embeddings where semantic similarity = vector proximity. You encode a document once, store its embedding, then find relevant context through fast vector operations—no need to re-encode pairs.

Key Takeaway: Sentence Transformers add pooling and contrastive training to base transformers, creating bi-encoders that independently embed text into semantically meaningful fixed-size vectors perfect for efficient retrieval.