Course contentsShow
← Machine Learning and Deep Learning
Lesson 1163 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

DistilBERT: Knowledge Distillation for Compression

Creating a smaller, faster BERT through distillation that retains 97% performance with 40% fewer parameters.

DistilBERT: Knowledge Distillation for Compression

You'll learn how to create a smaller, faster BERT model that keeps nearly all the performance while using far fewer resources.

The Core Problem

BERT is powerful but massive. BERT Base has 110 million parameters, making it slow and expensive to deploy in production. What if you could create a "student" model that learns from BERT's knowledge but is much smaller?

What Is Knowledge Distillation?

Think of it like a professor (BERT) teaching a bright undergraduate (DistilBERT). The student doesn't need to read every textbook the professor did—they can learn the condensed wisdom directly from the teacher.

DistilBERT uses three training objectives:

  1. Distillation loss: Learn to mimic BERT's output probability distributions (the "soft" predictions), not just hard labels
  2. Masked language modeling: Still learn the language task itself
  3. Cosine embedding loss: Match BERT's hidden state directions

The Architectural Changes

DistilBERT makes strategic cuts:

  • Removes token-type embeddings (no segment embeddings)
  • Halves the number of layers (6 instead of 12 for BERT Base)
  • Keeps the hidden dimension (768) to preserve representation capacity

The Remarkable Results

  • 40% fewer parameters (66M vs 110M)
  • 60% faster at inference
  • Retains 97% of BERT's performance on GLUE benchmark
  • 95% of BERT's performance on question answering

This isn't just making BERT smaller—it's teaching a compact model to be nearly as smart by learning from BERT's nuanced understanding rather than just the training data's hard labels.

Key Takeaway: DistilBERT proves that knowledge distillation can compress large pretrained models dramatically while preserving most of their capabilities, making deployment feasible on resource-constrained devices.