BERT Token Embeddings: Token, Segment, Position
What you'll learn: How BERT combines three distinct types of embeddings to create rich token representations that capture meaning, sentence boundaries, and position information.
The Three-Part Embedding System
BERT doesn't just look up a single embedding vector for each token. Instead, it combines three separate embeddings for every input token, creating a richer initial representation before any Transformer layers process it.
Think of it like describing a person at a party: you'd mention who they are (identity), which group they're with (segment), and where they're standing in line (position). BERT does the same for tokens.
The Three Embedding Types
Token Embeddings
These are the word/subword embeddings learned during pretraining. They capture the semantic meaning of each token (like "cat" or "running"). This is similar to Word2Vec or GloVe embeddings, but learned as part of BERT's training.
Segment Embeddings
BERT often processes two sentences at once (especially for Next Sentence Prediction). Segment embeddings mark which sentence each token belongs to—either Sentence A or Sentence B. This helps BERT understand sentence boundaries without explicit separators.
Position Embeddings
Unlike sinusoidal encodings in the original Transformer, BERT uses learned positional embeddings. These indicate each token's absolute position in the sequence (1st, 2nd, 3rd, etc.), allowing BERT to understand word order.
How They Combine
All three embeddings are simply added together element-wise to create the final input representation:
Final_Embedding = Token_Emb + Segment_Emb + Position_Emb
This sum goes into BERT's first Transformer encoder layer, giving it rich information about each token's identity, sentence membership, and sequential position from the very start.
Key Takeaway: BERT represents each input token by summing three learned embeddings—token (meaning), segment (sentence membership), and position (sequential order)—creating comprehensive initial representations for the Transformer stack.