Multilingual BERT and Cross-Lingual Transfer
What you'll learn: How a single BERT model can understand 100+ languages and transfer knowledge between them without explicit translation.
What Is Multilingual BERT?
Multilingual BERT (mBERT) is a variant of BERT trained on Wikipedia text from 104 languages simultaneously using the same masked language modeling objective you've already learned. Unlike domain-specific BERT models that specialize in one area, mBERT learns shared representations across languages—all within a single model with the same architecture as BERT Base.
The Cross-Lingual Transfer Magic
Here's where it gets exciting: zero-shot cross-lingual transfer. You can fine-tune mBERT on a task (like sentiment analysis) in one language—say English—and then apply it directly to another language like German or Chinese without any training data in those languages.
Analogy: Imagine learning to identify "happy" vs "sad" faces by looking at American movies. mBERT would then recognize those emotions in Japanese films without ever seeing Japanese examples during training—because it learned universal patterns of emotion, not language-specific ones.
How Training on 100+ Languages Works
mBERT uses the same WordPiece tokenization strategy across all languages with a shared vocabulary of ~110,000 subwords. During pretraining:
- Text from all 104 languages is mixed together
- The model learns to predict masked tokens across languages
- Languages with shared scripts, cognates, or linguistic features naturally develop overlapping representations
- The bidirectional context helps the model discover universal syntactic and semantic patterns
The model wasn't explicitly told which tokens belong to which language—it discovered cross-lingual alignments naturally through the pretraining objective.
Key takeaway: Multilingual BERT enables zero-shot cross-lingual transfer by learning shared multilingual representations during pretraining, allowing models trained on high-resource languages to work on low-resource languages without additional training data.