Image Patches as Tokens
What you'll learn: How Vision Transformers treat image regions like words by splitting images into fixed-size patches.
The Core Idea
In natural language processing, Transformers process text as a sequence of tokens (words or subwords). Vision Transformers (ViTs) apply the same principle to images by dividing them into fixed-size patches that serve as the basic input units—essentially treating each patch as a "visual token."
Think of it like cutting a photograph into a grid of equally-sized squares, similar to a jigsaw puzzle. Each square becomes one token that the Transformer processes, just like it would process individual words in a sentence.
How It Works
Instead of processing pixels individually (which would be computationally prohibitive), an image is split into non-overlapping patches:
- Divide the image: A 224×224 pixel image might be split into 16×16 pixel patches
- Create a grid: This produces a 14×14 grid (196 total patches)
- Flatten each patch: Each patch is converted into a vector
- Sequence of tokens: These 196 patch vectors become the input sequence to the Transformer
For example, with patch size 16×16:
- Each patch contains 16 × 16 × 3 = 768 values (for RGB images)
- The entire image becomes a sequence of patches, just like a sentence is a sequence of words
Why Patches Matter
Patches balance computational efficiency with spatial information. Too small, and you have too many tokens; too large, and you lose fine-grained detail. The patch size becomes a hyperparameter that affects both model performance and computational cost.
Key Takeaway: Vision Transformers convert images into sequences by dividing them into fixed-size patches, treating each patch as a token analogous to words in NLP, enabling Transformer architectures to process visual data.