Transfer Learning from Masked Models
You'll learn how to adapt pretrained masked image models like MAE and BEiT to solve specific downstream tasks.
The Core Idea
After pretraining a masked model (like MAE) on massive unlabeled datasets, you need to transfer that learned knowledge to your actual task—whether that's image classification, object detection, or segmentation. Two main strategies exist:
Fine-tuning means taking the pretrained encoder, attaching a new task-specific head (like a classification layer), and continuing to train the entire model (or parts of it) on your labeled dataset. You're adapting the learned representations to your specific problem.
Linear probing is a diagnostic approach: you freeze the pretrained encoder completely and train only a simple linear classifier on top of the extracted features. This tests how good the representations are "out of the box" without any adaptation.
Why Both Matter
Think of fine-tuning like learning a language, then reading specialized medical texts—you adjust your existing knowledge. Linear probing is like testing if your language skills transfer immediately to reading poetry without any additional study.
Linear probing tells you representation quality, while fine-tuning gives you best task performance. With masked models, you typically:
- Remove the decoder (it was only for pretraining reconstruction)
- Keep the encoder with its learned positional embeddings
- Add your task head (classifier, detection head, etc.)
- Choose: freeze encoder (linear probe) or train end-to-end (fine-tune)
For fine-tuning, lower learning rates for the pretrained encoder than the new head prevents catastrophic forgetting of useful representations.
Key Takeaway: Transfer learning from masked models involves either fine-tuning the pretrained encoder with a task-specific head for best performance, or linear probing (freezing the encoder) to evaluate representation quality—both strategies leverage the rich features learned during self-supervised pretraining.