Course contentsShow
← Machine Learning and Deep Learning
Lesson 1614 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Embedding and Unembedding Layers

Learn the role of input embeddings and output projection to vocabulary, including weight tying considerations.

Embedding and Unembedding Layers

What You'll Learn

You'll understand how large language models convert discrete tokens into continuous vectors and back, including the clever trick of weight tying.

The Problem: From Tokens to Vectors and Back

Language models work with words or subword tokens (discrete symbols), but neural networks process continuous vectors. The embedding layer converts token IDs into dense vector representations, while the unembedding layer (also called the output projection or LM head) converts the model's final hidden states back into vocabulary predictions.

How It Works

Think of embeddings as a lookup table: each token ID maps to a learned vector (often 768, 2048, or more dimensions). If your vocabulary has 50,000 tokens and embedding dimension is 2048, you have a [50000, 2048] matrix.

Input token ID: 42 → Embedding lookup → Vector [0.23, -0.15, 0.87, ...]

At the output, the unembedding layer projects the model's final hidden state (same dimension as embeddings) back to vocabulary size, producing logits for each possible next token. This is typically a linear layer: [hidden_dim → vocab_size].

Weight Tying: A Memory-Saving Trick

Modern LLMs often use weight tying: the embedding and unembedding layers share the same weight matrix (transposed for unembedding). Why? It reduces parameters by half for these layers and enforces a useful constraint—tokens with similar embeddings produce similar output probabilities.

Without weight tying: 50000 × 2048 × 2 = 204M parameters With weight tying: 50000 × 2048 = 102M parameters

This is especially valuable when vocabulary and hidden dimensions are large.

Key Takeaway: Embedding layers map tokens to dense vectors that the model processes, while unembedding layers project final states back to vocabulary logits; weight tying these layers saves memory and improves model coherence without sacrificing performance.