Self-Attention: Attending to the Same Sequence
What You'll Learn
You'll understand how self-attention allows each position in a sequence to attend to all other positions within the same sequence, rather than across two different sequences.
The Core Idea
So far, you've seen attention used between an encoder and decoder—two different sequences. Self-attention applies the same attention mechanism within a single sequence, allowing each element to "look at" and gather information from all other elements in that same sequence.
Think of it like a group discussion where every person listens to everyone else (including themselves) before forming their own understanding. Each word in a sentence considers the context provided by every other word to refine its own representation.
How It Works
In self-attention, the Query (Q), Key (K), and Value (V) vectors all come from the same input sequence. For a sequence of words:
- Each position generates its own Q, K, and V vectors
- Each position's Query attends to all positions' Keys (including its own)
- Attention scores reveal which positions are most relevant to each other
- The weighted sum of Values creates context-aware representations
For example, in "The animal didn't cross the street because it was too tired," self-attention helps "it" attend strongly to "animal" (not "street"), enriching its representation with relevant context.
Why This Matters
Self-attention captures dependencies between all positions simultaneously, regardless of distance. Unlike RNNs that process sequentially, self-attention computes relationships in parallel, making it both faster and better at capturing long-range dependencies.
Key Takeaway: Self-attention applies attention within a single sequence, allowing each position to dynamically gather context from all other positions—the foundation of the Transformer architecture.