Course contentsShow
← Machine Learning and Deep Learning
Lesson 2687 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Distilling Transformers and Language Models

Specialized techniques for compressing BERT-like models including DistilBERT architecture choices.

Distilling Transformers and Language Models

What you'll learn: How to compress massive transformer models like BERT into smaller, faster versions while preserving most of their language understanding capabilities.

The Challenge with Transformers

Modern language models like BERT contain hundreds of millions of parameters, making them slow and memory-intensive. Knowledge distillation offers a powerful solution: train a compact "student" transformer to mimic a large "teacher" model's behavior.

Specialized Techniques for Transformers

Unlike distilling CNNs for images, transformers require techniques tailored to their architecture:

Layer selection: Instead of matching every layer, you might distill only key attention patterns or final hidden states. The teacher's 12-layer BERT can guide a 6-layer student effectively.

Attention transfer: Transformers' self-attention weights capture linguistic relationships. You can add loss terms that encourage the student's attention distributions to match the teacher's—preserving how the model "looks at" different words.

Embedding alignment: The token embeddings and hidden representations can be explicitly aligned between teacher and student, even when dimensions differ.

DistilBERT: A Case Study

DistilBERT exemplifies these principles:

  • Reduce layers: 6 layers instead of 12
  • Wider hidden size: Kept 768 dimensions to preserve representational capacity
  • Triple loss: Combines distillation loss (soft targets), masked language modeling loss, and cosine embedding loss between hidden states
  • Result: 40% smaller, 60% faster, retains 97% of BERT's language understanding

The key insight: transformers are over-parameterized for many tasks, and a well-trained student can compress the teacher's knowledge into far fewer parameters.

Key Takeaway: Distilling transformers requires attention-aware and layer-wise strategies beyond standard distillation; DistilBERT demonstrates that careful architecture choices and multi-objective losses can dramatically reduce model size while maintaining strong performance.