Vision Transformer Encoder Architecture
What you'll learn: How Vision Transformers stack multiple transformer encoder blocks to process image patches and build rich visual representations.
The Core Idea
Now that we have image patches converted to tokens (with patch embeddings, positional embeddings, and a class token), we need to process them. The Vision Transformer uses the same encoder architecture proven in NLP: multiple stacked transformer encoder blocks, each containing multi-head self-attention and feedforward MLPs (Multi-Layer Perceptrons).
Think of it like a stack of filters that progressively refine your understanding. The first encoder block might learn to relate nearby patches (like edges), middle blocks might identify object parts, and deeper blocks capture high-level semantic relationships across the entire image.
How the Stack Works
Each encoder block consists of:
- Multi-head self-attention: Allows every patch token to "attend to" every other patch token, learning spatial relationships across the entire image
- MLP (feedforward network): Processes each token independently with non-linear transformations
- Residual connections and layer normalization: Stabilize training and enable gradient flow through many layers
The output tokens from one encoder block become the input to the next. After passing through all blocks (typically 12 or 24 layers), the final CLS token representation contains a rich summary of the entire image, ready for classification or other downstream tasks.
Unlike CNNs that gradually reduce spatial dimensions, Vision Transformers maintain the same number of tokens throughout the stack, letting global information flow freely from the very first layer.
Key Takeaway: Vision Transformer encoders stack multiple self-attention and MLP blocks to progressively refine patch relationships, enabling global image understanding rather than local receptive fields like CNNs.