Linear Projections for Queries, Keys, and Values
What you'll learn: How separate weight matrices transform inputs into Q, K, V for each attention head independently.
The Core Idea
In multi-head attention, we don't use the raw input embeddings directly as queries, keys, and values. Instead, each attention head learns its own linear projections — separate weight matrices that transform the input into specialized Q, K, and V representations.
Think of it like having multiple teams of translators, where each team has their own specialized dictionaries (weight matrices) to translate the same source text into different "dialects" that emphasize different aspects of meaning.
How It Works
For each attention head i, we have three distinct weight matrices:
- W^Q_i: Projects input → Query
- W^K_i: Projects input → Key
- W^V_i: Projects input → Value
Given an input embedding x (dimension d_model), each head computes:
Q_i = x @ W^Q_i
K_i = x @ W^K_i
V_i = x @ W^V_i
Each weight matrix has shape [d_model, d_k], where d_k is typically d_model / num_heads. This means each head focuses on a lower-dimensional "slice" of the full representation.
Why Separate Projections?
These learned projections allow each head to specialize in detecting different patterns or relationships. One head might learn to focus on positional relationships, another on semantic similarity, and another on syntactic structure — all from the same input.
The weights are learned during training through backpropagation, just like any other neural network parameters.
Key Takeaway: Linear projections use separate learned weight matrices (W^Q, W^K, W^V) to transform inputs into specialized query, key, and value representations for each attention head, enabling parallel learning of diverse attention patterns.