Course contentsShow
← Machine Learning and Deep Learning
Lesson 1084 of 3,53824. The Transformer ArchitectureFree lesson

Adding Positional Encodings to Token Embeddings

Learning how positional encodings combine with token embeddings before entering the Transformer layers.

Adding Positional Encodings to Token Embeddings

You'll learn: How positional encodings combine with token embeddings to give Transformers sequence order information.

The Core Idea

In Transformers, we have two types of information we need to represent:

  1. Token embeddings — what each word/token means
  2. Positional encodings — where each token sits in the sequence

These two pieces are combined through simple element-wise addition before the data enters the Transformer layers.

Why Addition Works

Think of it like labeling boxes in a warehouse. Each box has:

  • A description label (token embedding): "coffee beans"
  • A location tag (positional encoding): "shelf 3, position 7"

When you add these vectors together, the resulting representation contains both identity and location information simultaneously. The neural network learns to disentangle and use both aspects during training.

The Mechanics

Given:

  • Token embedding vector: e ∈ ℝ^d (e.g., d=512)
  • Positional encoding vector: p ∈ ℝ^d (same dimension)

The input to the first Transformer layer is simply:

input = e + p

Both vectors must have identical dimensions for this addition to work. The sinusoidal positional encoding you just learned generates vectors of exactly the same size as your token embeddings, making this combination straightforward.

Why Not Concatenation?

Addition preserves the model dimension and mixes positional information throughout the representation. Concatenation would double the vector size and keep position artificially separate — addition is more parameter-efficient and allows the attention mechanism to naturally blend both types of information.

Key Takeaway: Positional encodings are added element-wise to token embeddings before entering Transformer layers, creating input representations that contain both semantic meaning and sequence position in a single vector.