Course contentsShow
← Machine Learning and Deep Learning
Lesson 1161 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

ALBERT: Parameter Reduction Through Factorization

Cross-layer parameter sharing and embedding factorization techniques that dramatically reduce BERT's parameter count.

ALBERT: Parameter Reduction Through Factorization

You'll learn how ALBERT drastically cuts BERT's parameter count using two clever tricks while maintaining performance.

The Problem: BERT Is Huge

BERT models contain millions or billions of parameters, making them expensive to train and deploy. The vocabulary embeddings alone can consume enormous memory, and stacking multiple Transformer layers multiplies parameters quickly.

Two Key Innovations

1. Embedding Factorization

Instead of directly projecting vocabulary tokens into the hidden dimension (e.g., 768), ALBERT uses a two-step process:

Standard BERT approach: Vocabulary size × Hidden dimension (e.g., 30,000 × 768 = 23M parameters)

ALBERT's factorized approach:

  • First: Vocabulary → Small embedding (e.g., 30,000 × 128)
  • Then: Small embedding → Hidden dimension (e.g., 128 × 768)
  • Total: 30,000 × 128 + 128 × 768 = ~4M parameters

Analogy: Instead of having a direct highway from every town to every city (expensive!), build local roads to regional hubs, then highways from hubs to cities.

2. Cross-Layer Parameter Sharing

ALBERT shares the same attention and feedforward weights across all Transformer layers. Rather than having 12 unique layers with independent parameters, all 12 layers use identical weights.

Impact: Instead of 12× the parameters for 12 layers, you have just 1× the parameters, reused 12 times. Each layer still processes information differently because inputs differ, but the transformation is consistent.

Key Takeaway: ALBERT uses embedding factorization to separate vocabulary size from hidden dimension, and shares parameters across all layers, reducing BERT's parameter count by up to 18× while maintaining competitive performance through deeper information reuse.