Masking Strategy: 80-10-10 Rule
You'll learn why BERT doesn't simply replace all masked tokens with [MASK] during pretraining.
The Problem with Always Using [MASK]
When BERT learns through Masked Language Modeling, it needs to predict hidden words. The obvious approach would be: replace every masked token with [MASK]. But there's a catch—during fine-tuning and actual use, BERT never sees [MASK] tokens! This creates a mismatch between pretraining and real-world application.
The 80-10-10 Solution
BERT's creators solved this with a clever split. When a token is selected for masking (15% of all tokens), BERT applies three different treatments:
- 80% of the time: Replace with
[MASK]- Example: "The cat sat" → "The [MASK] sat"
- 10% of the time: Replace with a random word
- Example: "The cat sat" → "The banana sat"
- 10% of the time: Keep unchanged
- Example: "The cat sat" → "The cat sat"
Why This Works
Think of it like training for a test where you won't always see blanks. The 80% [MASK] tokens provide the main learning signal. The 10% random tokens force BERT to detect and correct nonsensical words, building robustness. The 10% unchanged tokens ensure BERT learns to process normal text and maintains representations even when no obvious error exists.
This strategy bridges the pretraining-finetuning gap while keeping the model alert to context rather than just memorizing that "whenever I see [MASK], predict something."
Key Takeaway: BERT's 80-10-10 masking rule (80% [MASK], 10% random, 10% unchanged) prevents overfitting to the [MASK] token and creates a more robust model that generalizes better to downstream tasks.