Course contentsShow
← Machine Learning and Deep Learning
Lesson 1182 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Domain Adaptation with Continued Pretraining

Continue pretraining BERT on domain-specific corpora before task fine-tuning for better performance.

Domain Adaptation with Continued Pretraining

You'll learn how to boost BERT's performance on specialized domains by continuing its pretraining on domain-specific text before fine-tuning.

The Core Idea

Imagine BERT as a general-purpose translator who learned from encyclopedias and novels. When you suddenly ask them to translate medical journals, they struggle with specialized terminology. Domain adaptation solves this by giving BERT a "crash course" in your domain's language before teaching it the final task.

Continued pretraining means taking a pretrained BERT model and running more masked language modeling (MLM) on domain-specific corpora—legal documents, scientific papers, medical records, or financial reports—before your task-specific fine-tuning.

Why This Works

Standard BERT was pretrained on general text (Wikipedia and books). When your downstream task involves specialized vocabulary, jargon, or writing patterns, there's a domain gap. The model hasn't seen enough domain-relevant context during pretraining.

By continuing pretraining on domain text:

  • The model adapts its word representations to domain-specific meanings
  • It learns domain vocabulary and writing patterns
  • It bridges the gap between general pretraining and your specialized task

The Three-Stage Process

  1. Base pretraining: BERT trains on general corpora (already done)
  2. Domain adaptation: Continue MLM on domain-specific unlabeled text
  3. Task fine-tuning: Fine-tune on your labeled task data

Think of it as: general education → domain specialization → job training.

When to Use It

Most valuable when your domain differs significantly from general text or uses specialized vocabulary. Less critical when working with general-domain tasks like sentiment analysis on product reviews.

Key Takeaway: Continued pretraining on domain-specific corpora creates a domain-adapted BERT that outperforms direct fine-tuning, especially when domain vocabulary and patterns differ significantly from general text.