Why Multiple Attention Heads?
What you'll learn: Why Transformers use multiple parallel attention mechanisms instead of just one, and how this captures richer patterns in the data.
The Limitation of Single Attention
A single attention mechanism computes one set of attention weights across the sequence. While powerful, it can only capture one type of relationship at a time. Think of it like looking at a sentence through a single lens—you might notice grammatical dependencies, but miss semantic similarities or positional patterns.
Multiple Perspectives on the Same Data
Multi-head attention runs several attention mechanisms in parallel, each with its own learned Query, Key, and Value weight matrices. This allows the model to attend to different aspects of the input simultaneously.
An Analogy
Imagine analyzing a photograph:
- One expert focuses on colors and lighting
- Another examines composition and framing
- A third studies the subjects and their emotions
Each brings a unique perspective. Similarly, different attention heads can specialize in different patterns: one might track syntactic relationships, another semantic meaning, and another positional information.
Representation Subspaces
Each attention head operates in its own representation subspace—a different learned transformation of the input. Rather than having one attention mechanism with dimension d_model, you might have 8 heads each working in dimension d_model/8. The outputs are then concatenated and projected back.
This parallel processing doesn't just add redundancy; it allows the model to simultaneously capture:
- Short-range vs long-range dependencies
- Syntactic vs semantic relationships
- Different linguistic phenomena in language
- Different visual patterns in images
Key Takeaway: Multiple attention heads allow the model to jointly attend to information from different representation subspaces, capturing diverse relationships that a single attention mechanism would miss—like having multiple experts analyze the same data simultaneously.