Multilingual Data Considerations
You'll learn how language balance, low-resource languages, and data mix decisions affect a large language model's multilingual capabilities.
The Challenge of Language Distribution
When building modern LLMs, deciding how much data to include from each language is far from straightforward. English dominates the web, making up 50-60% of quality text data. If you train purely on naturally-occurring proportions, your model becomes excellent at English but mediocre or poor at other languages.
Think of it like a classroom where the teacher spends 80% of time with students who already speak the language fluently—the others fall behind.
Balancing Act: Upsampling and Downsampling
Practitioners typically upsample (repeat or over-represent) data from low-resource languages—those with limited web presence like Swahili, Bengali, or Welsh. Simultaneously, they may downsample English to prevent it from overwhelming the model's capacity.
The goal: achieve reasonable performance across many languages without sacrificing too much English capability (since many downstream tasks still require it).
Low-Resource Language Challenges
Languages with minimal training data face fundamental obstacles:
- Limited vocabulary coverage in tokenizers
- Poor grammar and idiom learning due to insufficient examples
- Cross-lingual contamination where the model defaults to English mid-sentence
- Evaluation difficulties since benchmarks often don't exist
Even with upsampling, a language needs a critical mass—typically millions of documents—to develop strong capabilities.
Impact on Model Capabilities
Your data mix directly determines which languages "emerge" as the model scales. A model trained on 95% English won't suddenly become multilingual at larger sizes. The multilingual capacity must be built in from the data foundation, balanced against training efficiency and compute budgets.
Key Takeaway: Multilingual capability requires deliberate oversampling of low-resource languages and careful data balancing—models only learn what their training data distribution teaches them.