Course contentsShow
← Machine Learning and Deep Learning
Lesson 1143 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

BERT's Masked Language Modeling Objective

How BERT randomly masks input tokens and predicts them, enabling bidirectional context learning.

BERT's Masked Language Modeling Objective

What you'll learn: How BERT learns bidirectional context by randomly masking words and predicting them from surrounding context.

The Core Innovation

Unlike traditional language models that predict the next word (looking only left-to-right), BERT's Masked Language Modeling (MLM) objective lets the model learn from both directions simultaneously. Here's how it works:

During pretraining, BERT randomly selects 15% of input tokens and masks them. The model's job is to predict what the original masked tokens were, using context from both left and right sides.

How Masking Works

Think of it like a fill-in-the-blank exercise:

Original sentence: "The cat sat on the mat"
Masked input: "The cat [MASK] on the mat"
BERT's task: Predict "sat"

The masking strategy is clever:

  • 80% of selected tokens → replaced with [MASK]
  • 10% → replaced with a random word
  • 10% → kept unchanged

This prevents the model from only learning about [MASK] tokens and forces it to maintain representations for all tokens.

Why This Enables Bidirectional Learning

Traditional left-to-right models can't look ahead—they'd "cheat" by seeing the answer. By masking tokens, BERT creates a task where:

  • The model must use surrounding context (left AND right)
  • There's no information leakage
  • Every word benefits from full sentence context

This is why BERT embeddings are truly contextual—each word's representation depends on its entire sentence, not just what came before.

Key Takeaway: BERT's masked language modeling randomly hides 15% of input tokens and trains the model to predict them using bidirectional context, enabling richer contextual representations than unidirectional approaches.