Course contentsShow
← Machine Learning and Deep Learning
Lesson 1636 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Data Mix Ratios and Domain Balancing

Determining optimal ratios of web text, books, code, and other domains, and how mixing affects downstream capabilities.

Data Mix Ratios and Domain Balancing

What you'll learn: How carefully adjusting the proportions of different data types in pretraining affects what large language models can and cannot do.

The Recipe Matters

After filtering and deduplicating your pretraining corpus, you face a critical question: how much of each data type should you use? Your corpus might contain web text, books, scientific papers, code, and conversational data—but throwing them together equally often produces suboptimal results.

Why Ratios Matter

Different domains teach different capabilities:

  • Web text provides broad world knowledge and common language patterns
  • Books improve coherence and long-form reasoning
  • Code enhances logical reasoning and structured thinking
  • Scientific papers boost technical accuracy and formal reasoning

Training on imbalanced data creates imbalanced capabilities. A model trained 90% on web text might excel at casual conversation but struggle with code generation, even if code exists in the training set.

Finding the Right Mix

Modern LLMs use carefully tuned ratios discovered through experimentation. For example, you might train on:

  • 60% filtered web crawl
  • 20% books
  • 15% code repositories
  • 5% academic papers

These aren't arbitrary—teams measure downstream task performance (coding benchmarks, reasoning tests, general knowledge) and adjust ratios accordingly. Sometimes "upsampling" rare but valuable domains (repeating code data multiple times) improves specific capabilities without requiring more total data.

The Balancing Act

Think of it like nutrition: you need carbs, protein, and vegetables, but the optimal ratio depends on your goals. Too much of one domain can actually hurt other capabilities through overfitting to that style.

Key Takeaway: The ratio of different data domains in pretraining isn't just about representation—it directly shapes which capabilities emerge strongly in your model, making domain balancing a critical architectural decision alongside model size and compute budget.