The Scale and Composition of Pretraining Corpora
What you'll learn: How modern LLMs are trained on massive, diverse datasets measured in trillions of tokens.
Understanding the Scale
Modern large language models aren't trained on thousands or millions of examples—they're trained on trillions of tokens. A token is roughly a word or word piece, so we're talking about text corpora equivalent to millions of books. For context:
- GPT-3 (175B parameters): ~300 billion tokens
- LLaMA models: 1-2 trillion tokens
- Post-Chinchilla models: Often 2+ trillion tokens (following compute-optimal ratios)
This scale is necessary because of the scaling laws we've explored: more data generally means better performance, especially when paired with appropriate model size.
Data Source Composition
LLM pretraining corpora are carefully mixed from multiple sources:
Web text (60-80%): Crawled internet data like Common Crawl, filtered for quality. This provides broad knowledge but varies in accuracy.
Books (10-20%): Long-form text from digitized books. Teaches coherent long-range reasoning and narrative structure.
Code (5-15%): Programming repositories like GitHub. Surprisingly improves logical reasoning, not just coding ability.
Other sources: Academic papers, Wikipedia, conversational data—each contributing specialized knowledge.
Think of it like training a chef: web text is everyday recipes, books are culinary textbooks, code is precise measurement techniques, and papers are food science. The mixture matters as much as the total amount.
Why Mixture Matters
Different sources teach different capabilities. Too much web text might degrade formal reasoning. Too little code might limit structured problem-solving. Training teams iterate on these ratios, guided by downstream task performance.
Key Takeaway: Modern LLMs train on trillion-token corpora carefully blended from web text, books, code, and specialized sources—with composition ratios fine-tuned to balance broad knowledge with specific capabilities.