Adding Positional Encodings to Token Embeddings
You'll learn: How positional encodings combine with token embeddings to give Transformers sequence order information.
The Core Idea
In Transformers, we have two types of information we need to represent:
- Token embeddings — what each word/token means
- Positional encodings — where each token sits in the sequence
These two pieces are combined through simple element-wise addition before the data enters the Transformer layers.
Why Addition Works
Think of it like labeling boxes in a warehouse. Each box has:
- A description label (token embedding): "coffee beans"
- A location tag (positional encoding): "shelf 3, position 7"
When you add these vectors together, the resulting representation contains both identity and location information simultaneously. The neural network learns to disentangle and use both aspects during training.
The Mechanics
Given:
- Token embedding vector: e ∈ ℝ^d (e.g., d=512)
- Positional encoding vector: p ∈ ℝ^d (same dimension)
The input to the first Transformer layer is simply:
input = e + p
Both vectors must have identical dimensions for this addition to work. The sinusoidal positional encoding you just learned generates vectors of exactly the same size as your token embeddings, making this combination straightforward.
Why Not Concatenation?
Addition preserves the model dimension and mixes positional information throughout the representation. Concatenation would double the vector size and keep position artificially separate — addition is more parameter-efficient and allows the attention mechanism to naturally blend both types of information.
Key Takeaway: Positional encodings are added element-wise to token embeddings before entering Transformer layers, creating input representations that contain both semantic meaning and sequence position in a single vector.