Sentence Transformers Architecture
What you'll learn: How sentence-level encoders create fixed-size embeddings using pooling strategies over token embeddings.
The Challenge: From Many Tokens to One Vector
When you encode a sentence like "The cat sat on the mat" with a transformer model (like BERT), you get individual embeddings for each token—typically one vector per word or subword. But for tasks like dense retrieval (which you just learned about), you need a single vector representing the entire sentence. How do you compress all those token embeddings into one meaningful representation?
Pooling Strategies: Combining Token Embeddings
Sentence Transformers solve this by applying a pooling layer after the transformer encoder. Think of pooling like taking a class photo: you have many individual students (tokens), but you want one picture (sentence embedding) that captures the whole group.
Common Pooling Methods
-
Mean Pooling: Average all token embeddings element-wise. This treats every word equally—like calculating the class average score.
-
CLS Token Pooling: Use only the special
[CLS]token's embedding (first token in BERT). This token is trained to summarize the whole sequence. -
Max Pooling: Take the maximum value across each dimension. Captures the strongest signals from any token.
The architecture flows: Input Text → Tokenization → Transformer Encoder → Pooling Layer → Fixed-Size Sentence Embedding.
These sentence embeddings are particularly powerful for semantic similarity tasks—sentences with similar meanings produce nearby vectors in embedding space.
Key Takeaway: Sentence Transformers create fixed-size sentence embeddings by applying pooling operations (mean, CLS, or max) over the variable-length token embeddings produced by transformer encoders, enabling efficient sentence-level comparisons.