Data Mix Ratios and Domain Balancing
What you'll learn: How carefully adjusting the proportions of different data types in pretraining affects what large language models can and cannot do.
The Recipe Matters
After filtering and deduplicating your pretraining corpus, you face a critical question: how much of each data type should you use? Your corpus might contain web text, books, scientific papers, code, and conversational data—but throwing them together equally often produces suboptimal results.
Why Ratios Matter
Different domains teach different capabilities:
- Web text provides broad world knowledge and common language patterns
- Books improve coherence and long-form reasoning
- Code enhances logical reasoning and structured thinking
- Scientific papers boost technical accuracy and formal reasoning
Training on imbalanced data creates imbalanced capabilities. A model trained 90% on web text might excel at casual conversation but struggle with code generation, even if code exists in the training set.
Finding the Right Mix
Modern LLMs use carefully tuned ratios discovered through experimentation. For example, you might train on:
- 60% filtered web crawl
- 20% books
- 15% code repositories
- 5% academic papers
These aren't arbitrary—teams measure downstream task performance (coding benchmarks, reasoning tests, general knowledge) and adjust ratios accordingly. Sometimes "upsampling" rare but valuable domains (repeating code data multiple times) improves specific capabilities without requiring more total data.
The Balancing Act
Think of it like nutrition: you need carbs, protein, and vegetables, but the optimal ratio depends on your goals. Too much of one domain can actually hurt other capabilities through overfitting to that style.
Key Takeaway: The ratio of different data domains in pretraining isn't just about representation—it directly shapes which capabilities emerge strongly in your model, making domain balancing a critical architectural decision alongside model size and compute budget.