Course contentsShow
← Machine Learning and Deep Learning
Lesson 1150 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Masking Strategy: 80-10-10 Rule

Why BERT masks tokens with 80% [MASK], 10% random tokens, and 10% unchanged tokens.

Masking Strategy: 80-10-10 Rule

You'll learn why BERT doesn't simply replace all masked tokens with [MASK] during pretraining.

The Problem with Always Using [MASK]

When BERT learns through Masked Language Modeling, it needs to predict hidden words. The obvious approach would be: replace every masked token with [MASK]. But there's a catch—during fine-tuning and actual use, BERT never sees [MASK] tokens! This creates a mismatch between pretraining and real-world application.

The 80-10-10 Solution

BERT's creators solved this with a clever split. When a token is selected for masking (15% of all tokens), BERT applies three different treatments:

  • 80% of the time: Replace with [MASK]
    • Example: "The cat sat" → "The [MASK] sat"
  • 10% of the time: Replace with a random word
    • Example: "The cat sat" → "The banana sat"
  • 10% of the time: Keep unchanged
    • Example: "The cat sat" → "The cat sat"

Why This Works

Think of it like training for a test where you won't always see blanks. The 80% [MASK] tokens provide the main learning signal. The 10% random tokens force BERT to detect and correct nonsensical words, building robustness. The 10% unchanged tokens ensure BERT learns to process normal text and maintains representations even when no obvious error exists.

This strategy bridges the pretraining-finetuning gap while keeping the model alert to context rather than just memorizing that "whenever I see [MASK], predict something."

Key Takeaway: BERT's 80-10-10 masking rule (80% [MASK], 10% random, 10% unchanged) prevents overfitting to the [MASK] token and creates a more robust model that generalizes better to downstream tasks.