Inductive Bias Differences
What you'll learn: How Vision Transformers rely on fundamentally different assumptions than CNNs, trading built-in structural bias for data-hungry flexibility.
The Core Difference
Inductive bias refers to the assumptions a model architecture makes about the data before seeing it. CNNs come pre-loaded with strong assumptions: locality (nearby pixels matter most), translation invariance (a cat is a cat whether it's in the corner or center), and spatial hierarchy (edges → textures → parts → objects). These biases are baked into the architecture through convolutional filters and pooling.
Vision Transformers, by contrast, start with minimal assumptions. The self-attention mechanism treats every patch equally—a patch from the top-left can attend to any patch anywhere else from layer one. There's no built-in notion that neighboring patches should interact more than distant ones. The positional embeddings provide spatial awareness, but they're learned, not hardcoded.
Why This Matters
CNNs: Strong inductive bias = sample efficient but potentially limiting. They work well even with smaller datasets because they "know" to look for local patterns.
ViTs: Weak inductive bias = need massive data to learn what CNNs assume. Given enough examples (think ImageNet-21k or JFT-300M), ViTs discover the right patterns themselves—and sometimes find patterns CNNs miss.
Think of it like teaching someone to cook: CNNs are given a detailed recipe book (strong bias), while ViTs start with just ingredients and learn by trying millions of combinations (data-driven learning).
Key Takeaway: Vision Transformers sacrifice CNNs' helpful built-in assumptions about images, requiring vastly more training data but gaining the flexibility to learn richer, more general representations when that data is available.