Positional Embeddings for 2D Grids
What you'll learn: How Vision Transformers preserve the spatial layout of image patches through positional information.
The Challenge with Flattened Patches
Remember how we converted image patches into tokens? Once those patches are flattened into a sequence, the model loses all information about where each patch was originally located in the 2D image grid.
Imagine cutting up a photograph into squares, shuffling them, then asking someone to describe what's in the picture. Without knowing which piece belongs where, spatial relationships (like "sky above mountains" or "eyes next to nose") become invisible.
Adding Position Awareness
Positional embeddings solve this by attaching location information to each patch embedding. Unlike 1D sequences (text), images have two-dimensional spatial structure—both row and column positions matter.
There are two main approaches:
Learnable 2D Positional Embeddings
The model learns a unique embedding vector for each position in the grid during training. If your image is divided into a 14×14 grid (196 patches), you'd have 196 learnable position vectors that get added to the corresponding patch embeddings.
Fixed Positional Encodings
Some models use mathematical functions (like sine/cosine waves at different frequencies) to generate position encodings, similar to the original Transformer but adapted for 2D coordinates.
Why This Matters
Without positional embeddings, a Vision Transformer would treat an image of an upside-down face exactly the same as a right-side-up face—the same patches exist, just in different locations. Positional information lets the model understand that "top-left" is different from "bottom-right."
Key Takeaway: Positional embeddings restore spatial awareness by adding learnable or fixed coordinate information to each patch, allowing Vision Transformers to understand the 2D layout of images despite processing patches as a flat sequence.