Course contentsShow
← Machine Learning and Deep Learning
Lesson 1326 of 3,53829. Core NLP TasksFree lesson

Sentence Transformers Architecture

Understand how sentence-level encoders create fixed-size embeddings using pooling strategies over token embeddings.

Sentence Transformers Architecture

What you'll learn: How sentence-level encoders create fixed-size embeddings using pooling strategies over token embeddings.

The Challenge: From Many Tokens to One Vector

When you encode a sentence like "The cat sat on the mat" with a transformer model (like BERT), you get individual embeddings for each token—typically one vector per word or subword. But for tasks like dense retrieval (which you just learned about), you need a single vector representing the entire sentence. How do you compress all those token embeddings into one meaningful representation?

Pooling Strategies: Combining Token Embeddings

Sentence Transformers solve this by applying a pooling layer after the transformer encoder. Think of pooling like taking a class photo: you have many individual students (tokens), but you want one picture (sentence embedding) that captures the whole group.

Common Pooling Methods

  1. Mean Pooling: Average all token embeddings element-wise. This treats every word equally—like calculating the class average score.

  2. CLS Token Pooling: Use only the special [CLS] token's embedding (first token in BERT). This token is trained to summarize the whole sequence.

  3. Max Pooling: Take the maximum value across each dimension. Captures the strongest signals from any token.

The architecture flows: Input Text → Tokenization → Transformer Encoder → Pooling Layer → Fixed-Size Sentence Embedding.

These sentence embeddings are particularly powerful for semantic similarity tasks—sentences with similar meanings produce nearby vectors in embedding space.

Key Takeaway: Sentence Transformers create fixed-size sentence embeddings by applying pooling operations (mean, CLS, or max) over the variable-length token embeddings produced by transformer encoders, enabling efficient sentence-level comparisons.