BERT's Masked Language Modeling Objective
What you'll learn: How BERT learns bidirectional context by randomly masking words and predicting them from surrounding context.
The Core Innovation
Unlike traditional language models that predict the next word (looking only left-to-right), BERT's Masked Language Modeling (MLM) objective lets the model learn from both directions simultaneously. Here's how it works:
During pretraining, BERT randomly selects 15% of input tokens and masks them. The model's job is to predict what the original masked tokens were, using context from both left and right sides.
How Masking Works
Think of it like a fill-in-the-blank exercise:
Original sentence: "The cat sat on the mat"
Masked input: "The cat [MASK] on the mat"
BERT's task: Predict "sat"
The masking strategy is clever:
- 80% of selected tokens → replaced with
[MASK] - 10% → replaced with a random word
- 10% → kept unchanged
This prevents the model from only learning about [MASK] tokens and forces it to maintain representations for all tokens.
Why This Enables Bidirectional Learning
Traditional left-to-right models can't look ahead—they'd "cheat" by seeing the answer. By masking tokens, BERT creates a task where:
- The model must use surrounding context (left AND right)
- There's no information leakage
- Every word benefits from full sentence context
This is why BERT embeddings are truly contextual—each word's representation depends on its entire sentence, not just what came before.
Key Takeaway: BERT's masked language modeling randomly hides 15% of input tokens and trains the model to predict them using bidirectional context, enabling richer contextual representations than unidirectional approaches.