Course contentsShow
← Machine Learning and Deep Learning
Lesson 1342 of 3,53830. Vision TransformersFree lesson

Vision Transformer Encoder Architecture

Stacking multiple transformer encoder blocks with multi-head self-attention and MLPs.

Vision Transformer Encoder Architecture

What you'll learn: How Vision Transformers stack multiple transformer encoder blocks to process image patches and build rich visual representations.

The Core Idea

Now that we have image patches converted to tokens (with patch embeddings, positional embeddings, and a class token), we need to process them. The Vision Transformer uses the same encoder architecture proven in NLP: multiple stacked transformer encoder blocks, each containing multi-head self-attention and feedforward MLPs (Multi-Layer Perceptrons).

Think of it like a stack of filters that progressively refine your understanding. The first encoder block might learn to relate nearby patches (like edges), middle blocks might identify object parts, and deeper blocks capture high-level semantic relationships across the entire image.

How the Stack Works

Each encoder block consists of:

  1. Multi-head self-attention: Allows every patch token to "attend to" every other patch token, learning spatial relationships across the entire image
  2. MLP (feedforward network): Processes each token independently with non-linear transformations
  3. Residual connections and layer normalization: Stabilize training and enable gradient flow through many layers

The output tokens from one encoder block become the input to the next. After passing through all blocks (typically 12 or 24 layers), the final CLS token representation contains a rich summary of the entire image, ready for classification or other downstream tasks.

Unlike CNNs that gradually reduce spatial dimensions, Vision Transformers maintain the same number of tokens throughout the stack, letting global information flow freely from the very first layer.

Key Takeaway: Vision Transformer encoders stack multiple self-attention and MLP blocks to progressively refine patch relationships, enabling global image understanding rather than local receptive fields like CNNs.