Fine-Tuning vs Feature Extraction with Contextual Embeddings
You'll learn when to freeze pretrained contextual embeddings versus updating them during training.
The Two Approaches
When using pretrained models like BERT or GPT, you face a crucial decision: feature extraction or fine-tuning.
Feature extraction treats the pretrained model as a frozen feature generator. You extract contextual embeddings from specific layers and feed them into a new classifier you train from scratch. The pretrained weights never change—like using a professional photographer's camera settings exactly as-is and just choosing which photos to print.
Fine-tuning updates the entire model end-to-end. Both the pretrained weights and your new classifier learn together during training. This is like adjusting the camera settings slightly for your specific lighting conditions while also learning what to photograph.
When to Use Each Approach
Choose feature extraction when:
- You have very limited training data (< 1,000 examples)
- Computational resources are constrained
- Your task is similar to the pretraining objective
- You want faster training and simpler deployment
Choose fine-tuning when:
- You have sufficient training data (thousands+ of examples)
- Your task differs significantly from pretraining
- You need maximum performance
- You have GPU resources available
The Practical Trade-off
Feature extraction is safer with small datasets—fine-tuning all parameters risks overfitting when you lack examples. However, fine-tuning lets the model adapt its contextual representations to your specific task's nuances, potentially achieving better performance.
Many practitioners start with feature extraction for quick experiments, then fine-tune if performance plateaus and more data becomes available.
Key Takeaway: Use feature extraction when data or resources are limited and your task aligns with pretraining; fine-tune when you have sufficient data and need the model to adapt its representations to your specific task.