Course contentsShow
← Machine Learning and Deep Learning
Lesson 1343 of 3,53830. Vision TransformersFree lesson

Multi-Head Self-Attention in ViT

How self-attention enables each patch to attend to all other patches globally.

Multi-Head Self-Attention in ViT

What you'll learn: How self-attention enables each patch to attend to all other patches globally in Vision Transformers.

The Global View Advantage

In traditional CNNs, each pixel or feature only "sees" nearby information through small receptive fields. Vision Transformers break this limitation with self-attention: every image patch can directly interact with every other patch, regardless of distance.

Think of it like a classroom discussion where every student can hear and respond to every other student simultaneously, rather than only talking to their immediate neighbors.

How Multi-Head Attention Works in ViT

After your image is divided into patches (say, 196 patches from a 224×224 image) and embedded with positional information, the self-attention mechanism computes three vectors for each patch:

  • Query (Q): "What am I looking for?"
  • Key (K): "What information do I contain?"
  • Value (V): "What information do I provide?"

Each patch's Query compares against all other patches' Keys to determine attention scores—essentially deciding which patches are most relevant to attend to. These scores weight the Values, creating a context-aware representation that incorporates information from the entire image.

The "multi-head" part means this process happens multiple times in parallel (typically 12 heads in ViT), each learning to focus on different relationships: one head might capture spatial proximity, another might detect color patterns, and another might recognize semantic relationships.

Why This Matters for Vision

Unlike CNNs that build global understanding gradually through layers, ViT's self-attention provides immediate global context. A patch from a cat's ear can instantly attend to patches containing the tail, eyes, or background—enabling holistic reasoning from the very first layer.

Key Takeaway: Multi-head self-attention in ViT allows every image patch to dynamically attend to all other patches simultaneously, creating globally-informed representations that capture long-range dependencies in a single operation.