Extracting and Using BERT Embeddings
What you'll learn: How to extract token-level, sentence-level, and CLS embeddings from BERT and use them in your own models.
The Core Idea
BERT generates contextual representations for every token in your input, but you need to know where and how to extract these embeddings for your specific task. Think of BERT as a sophisticated feature extractor: you feed in text, and it gives you rich numerical representations at multiple levels—individual words, whole sentences, or special summary tokens.
Types of Embeddings You Can Extract
Token Embeddings
Each word (or subword) gets its own contextual vector. Extract these from any layer (often the last 4 layers work best) to capture word meaning in context. Use them for tasks like named entity recognition or token classification.
CLS Token Embedding
BERT adds a special [CLS] token at the start of every input. Its final embedding serves as a sentence-level representation, designed during pretraining to capture overall meaning. Extract this single vector for sentence classification tasks.
Sentence Embeddings
For a fixed-length representation of variable-length text, you can average all token embeddings (excluding padding) or use the CLS token. This gives you one vector per sentence for tasks like similarity comparison or clustering.
Common Extraction Pattern
You typically freeze BERT's weights and use its output as features, or fine-tune it while extracting embeddings. The key is choosing the right layer: deeper layers capture more task-specific patterns, while middle layers often contain the most generalizable semantic information.
Key Takeaway: BERT embeddings come in three flavors—token-level for per-word tasks, CLS for sentence classification, and averaged pooling for sentence similarity—each extracted from specific positions and layers depending on your downstream needs.