Course contentsShow
← Machine Learning and Deep Learning
Lesson 1183 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Catastrophic Forgetting and Regularization

Prevent loss of pretrained knowledge during fine-tuning using techniques like mixout and adapter layers.

Catastrophic Forgetting and Regularization

You'll learn how to protect BERT's pretrained knowledge from being overwritten during fine-tuning.

The Problem: Catastrophic Forgetting

When you fine-tune a pretrained model like BERT on a specific task, you risk catastrophic forgetting—the model "overwrites" its valuable pretrained knowledge with task-specific patterns. It's like a seasoned chef who, after working exclusively on desserts for weeks, suddenly forgets how to make basic sauces.

BERT spent enormous compute learning general language understanding from massive datasets. If fine-tuning is too aggressive, the model's weights shift dramatically, losing that broad knowledge and becoming overly specialized—performing well on your task but poorly on general language understanding.

Regularization Techniques to Prevent Forgetting

Mixout is a dropout-inspired technique that randomly keeps some weights at their pretrained values during fine-tuning. Think of it as anchoring parts of the model to its original knowledge while allowing other parts to adapt.

Adapter layers add small, trainable modules between frozen pretrained layers. Instead of modifying BERT's weights, you insert compact layers that learn task-specific transformations. The pretrained knowledge stays intact while adapters handle specialization—like adding specialized tools to a chef's kitchen without changing their fundamental skills.

Elastic Weight Consolidation (EWC) penalizes changes to weights that were important for pretraining, allowing less critical weights to adapt more freely.

These techniques are especially crucial when:

  • Fine-tuning on small datasets (higher overfitting risk)
  • Planning to use the model across multiple tasks
  • Needing to maintain general language capabilities

Key Takeaway: Catastrophic forgetting erases pretrained knowledge during fine-tuning; use regularization techniques like mixout or adapter layers to preserve BERT's general understanding while adapting to your specific task.