Resolution and Patch Size Trade-offs
What you'll learn: How changing patch size in Vision Transformers affects the number of tokens, computational cost, and model performance.
The Fundamental Trade-off
When you split an image into patches for a Vision Transformer, you face a critical decision: how big should each patch be? This seemingly simple choice has cascading effects on everything from memory usage to accuracy.
The Sequence Length Connection
Remember that each patch becomes one token in your sequence. For an image of resolution 224×224:
- 16×16 patches → (224/16)² = 196 tokens
- 32×32 patches → (224/32)² = 49 tokens
- 8×8 patches → (224/8)² = 784 tokens
Since self-attention has quadratic complexity in sequence length, smaller patches dramatically increase computation. Doubling the resolution (halving patch size) quadruples the number of tokens!
The Performance Perspective
Smaller patches capture finer visual details—think of them as higher "resolution tokens." This helps the model see subtle textures and edges. However, this comes at the cost of much longer sequences and slower training.
Larger patches are computationally cheaper but may miss fine-grained patterns. They work better when you have limited data or compute, but might underperform on tasks requiring detailed visual understanding.
Resolution Flexibility
Interestingly, you can adjust input image resolution independently of patch size. A 384×384 image with 16×16 patches gives you 576 tokens—more detail than 224×224, but requiring careful positional embedding interpolation (remember those 2D grids?).
Key Takeaway: Patch size is a critical hyperparameter that balances computational efficiency against the model's ability to capture fine visual details—smaller patches mean richer representations but dramatically higher costs.