Course contentsShow
← Machine Learning and Deep Learning
Lesson 1057 of 3,53824. The Transformer ArchitectureFree lesson

Self-Attention: Attending to the Same Sequence

Understand how self-attention allows each position to attend to all positions in the same sequence.

Self-Attention: Attending to the Same Sequence

What You'll Learn

You'll understand how self-attention allows each position in a sequence to attend to all other positions within the same sequence, rather than across two different sequences.

The Core Idea

So far, you've seen attention used between an encoder and decoder—two different sequences. Self-attention applies the same attention mechanism within a single sequence, allowing each element to "look at" and gather information from all other elements in that same sequence.

Think of it like a group discussion where every person listens to everyone else (including themselves) before forming their own understanding. Each word in a sentence considers the context provided by every other word to refine its own representation.

How It Works

In self-attention, the Query (Q), Key (K), and Value (V) vectors all come from the same input sequence. For a sequence of words:

  1. Each position generates its own Q, K, and V vectors
  2. Each position's Query attends to all positions' Keys (including its own)
  3. Attention scores reveal which positions are most relevant to each other
  4. The weighted sum of Values creates context-aware representations

For example, in "The animal didn't cross the street because it was too tired," self-attention helps "it" attend strongly to "animal" (not "street"), enriching its representation with relevant context.

Why This Matters

Self-attention captures dependencies between all positions simultaneously, regardless of distance. Unlike RNNs that process sequentially, self-attention computes relationships in parallel, making it both faster and better at capturing long-range dependencies.

Key Takeaway: Self-attention applies attention within a single sequence, allowing each position to dynamically gather context from all other positions—the foundation of the Transformer architecture.