Course contentsShow
← Machine Learning and Deep Learning
Lesson 1173 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Fine-Tuning vs Feature Extraction

Compare fine-tuning BERT end-to-end versus freezing layers and using BERT as a feature extractor.

Fine-Tuning vs Feature Extraction

What you'll learn: Two fundamentally different ways to use pretrained BERT models—updating all weights versus freezing the model and using it as a fixed feature encoder.

The Core Trade-off

When you have a pretrained BERT model (you've learned about BERT, RoBERTa, ALBERT, and others), you face a choice: should you fine-tune the entire model or use it as a feature extractor?

Fine-Tuning: The Full Update Approach

Fine-tuning means you take the pretrained BERT weights and continue training them on your specific task. All layers (or most of them) get updated through backpropagation. You're essentially teaching BERT to adapt its contextual representations to your particular problem—whether that's sentiment analysis, question answering, or named entity recognition.

Think of it like hiring an experienced chef (pretrained BERT) and letting them adjust their cooking style to match your restaurant's menu. They bring expertise but adapt to your needs.

Feature Extraction: The Frozen Approach

Feature extraction means you freeze BERT's weights completely. You pass your text through BERT to get contextual embeddings (remember those layer-wise representations?), then feed those embeddings into a new classifier head that you train from scratch. BERT acts purely as a sophisticated feature generator.

This is like hiring that same chef but only to prep ingredients—they chop, measure, and organize everything using their expertise, but someone else does the actual cooking.

When to Use Which?

  • Fine-tune when you have sufficient labeled data (thousands+ examples) and want maximum performance
  • Feature extract when you have limited data, want faster training, need lower memory, or want to avoid catastrophic forgetting of BERT's general knowledge

Key Takeaway: Fine-tuning updates BERT's weights for task-specific adaptation (higher performance, more resources), while feature extraction freezes BERT as a fixed contextual encoder (faster, safer, but potentially less accurate).