BERT's Attention Patterns: What They Learn
You'll learn: How BERT's different attention heads specialize to capture distinct linguistic phenomena, from syntactic relationships to semantic meaning.
The Magic Inside BERT's Heads
Remember that BERT has multiple attention heads across its layers? It turns out these heads don't all learn the same thing—they specialize in capturing different aspects of language.
Think of it like a team of editors reviewing a document: one focuses on grammar, another on word meanings, another on sentence structure, and so on. Each attention head in BERT develops its own "specialty" during pretraining.
What Researchers Have Discovered
Through visualization and analysis of attention weights, researchers found striking patterns:
Syntactic heads learn grammatical structure—one head might connect verbs to their subjects, another links pronouns to their antecedents, and another tracks dependency relationships (like which words modify which).
Positional heads focus on relative word positions, often attending to adjacent words or specific offsets (like "the word three positions back").
Semantic heads capture meaning relationships—synonyms, related concepts, or words that co-occur in similar contexts.
Delimiter heads pay special attention to separator tokens like [SEP] and [CLS], helping distinguish between sentence segments.
Why This Matters
This specialization happens automatically during masked language modeling—BERT wasn't explicitly told to learn syntax or semantics separately. The multi-head architecture naturally divides labor, with different heads discovering useful patterns that together help predict masked words.
Lower layers tend to capture more surface-level syntax, while deeper layers learn increasingly abstract semantic relationships.
Key Takeaway: BERT's attention heads spontaneously develop specialized roles during pretraining, with different heads learning to capture syntax, semantics, positional patterns, and structural markers—all without explicit supervision.