From CNNs to Vision Transformers
What you'll learn: Why the field is shifting from CNNs to transformers for vision tasks, and what fundamentally changes in that transition.
The CNN Era
For years, Convolutional Neural Networks (CNNs) dominated computer vision. They work by sliding filters across images to detect local patterns—edges, textures, shapes—building up hierarchical representations through layers. CNNs have inductive biases baked in: locality (nearby pixels matter more) and translation invariance (a cat is a cat whether it's left or right).
Why Transformers for Vision?
Transformers, originally designed for text, excel at capturing long-range dependencies through self-attention. In NLP, attention lets models relate words regardless of distance. The insight: images can be treated as sequences too.
Instead of pixels, we divide an image into patches (like 16×16 grids), flatten each patch into a vector, and feed them as tokens to a transformer. Now the model can relate any patch to any other patch directly—no need to stack many layers to "see" across the image.
Key Differences from CNNs
- No built-in locality bias: Transformers don't assume nearby patches are related; they learn relationships from data
- Global receptive field from layer 1: Every patch can attend to every other patch immediately
- Scalability: Transformers scale better with more data and parameters, often outperforming CNNs when trained on massive datasets
- Flexibility: Same architecture works for multiple modalities (text, images, video)
Key Takeaway: Vision Transformers trade CNNs' built-in spatial assumptions for learned attention mechanisms that capture global context from the start, excelling when abundant data is available.