Patch Embedding Layer
What you'll learn: How Vision Transformers convert image patches into vector representations that can be processed like word tokens in language models.
The Core Idea
In Vision Transformers, we first split an image into fixed-size patches (like 16×16 pixels), treating each patch as a "token"—similar to how words are tokens in NLP. But Transformers can't directly process raw pixel grids; they need fixed-length vectors.
The Patch Embedding Layer solves this by flattening each patch into a 1D vector and then applying a linear projection (a learnable matrix multiplication) to map it into an embedding vector of a chosen dimension (often 768 or 1024).
Why This Matters
Think of it like converting different-sized photos into standard passport photos. No matter what's in the original patch, it gets transformed into a uniform vector representation that the Transformer can work with.
The process:
- Flatten: A 16×16×3 RGB patch becomes a vector of length 768 (16 × 16 × 3)
- Project: Multiply by a learned weight matrix to produce the embedding dimension (e.g., 768 → 768)
This projection is typically implemented as a single convolutional layer with kernel size equal to the patch size and stride equal to the patch size—it simultaneously splits the image into patches and embeds them.
What Gets Learned
The weight matrix in this layer learns to extract meaningful features from raw pixel values, similar to how word embeddings learn semantic meaning. Different dimensions might capture edges, colors, textures, or more abstract visual concepts.
Key Takeaway: The Patch Embedding Layer linearly projects flattened image patches into fixed-dimensional embedding vectors, converting visual data into the token-like format that Transformers require.