Embedding and Unembedding Layers
What You'll Learn
You'll understand how large language models convert discrete tokens into continuous vectors and back, including the clever trick of weight tying.
The Problem: From Tokens to Vectors and Back
Language models work with words or subword tokens (discrete symbols), but neural networks process continuous vectors. The embedding layer converts token IDs into dense vector representations, while the unembedding layer (also called the output projection or LM head) converts the model's final hidden states back into vocabulary predictions.
How It Works
Think of embeddings as a lookup table: each token ID maps to a learned vector (often 768, 2048, or more dimensions). If your vocabulary has 50,000 tokens and embedding dimension is 2048, you have a [50000, 2048] matrix.
Input token ID: 42 → Embedding lookup → Vector [0.23, -0.15, 0.87, ...]
At the output, the unembedding layer projects the model's final hidden state (same dimension as embeddings) back to vocabulary size, producing logits for each possible next token. This is typically a linear layer: [hidden_dim → vocab_size].
Weight Tying: A Memory-Saving Trick
Modern LLMs often use weight tying: the embedding and unembedding layers share the same weight matrix (transposed for unembedding). Why? It reduces parameters by half for these layers and enforces a useful constraint—tokens with similar embeddings produce similar output probabilities.
Without weight tying: 50000 × 2048 × 2 = 204M parameters
With weight tying: 50000 × 2048 = 102M parameters
This is especially valuable when vocabulary and hidden dimensions are large.
Key Takeaway: Embedding layers map tokens to dense vectors that the model processes, while unembedding layers project final states back to vocabulary logits; weight tying these layers saves memory and improves model coherence without sacrificing performance.