Course contentsShow
← Machine Learning and Deep Learning
Lesson 1637 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

The Role of Code in Pretraining

Why including code improves reasoning abilities, typical code corpus sources like GitHub, and code-to-text ratios in practice.

The Role of Code in Pretraining

You'll learn why training language models on code dramatically improves their reasoning abilities and how practitioners balance code with natural language text.

Why Code Improves Reasoning

Including programming code in pretraining corpora has become standard practice—not just for building coding assistants, but because code makes models smarter overall. Code is structured, logical, and forces precise thinking. When models learn from code, they absorb patterns of:

  • Formal reasoning: Functions must produce correct outputs given inputs
  • Compositional structure: Complex solutions built from simple components
  • Explicit logic: If-then patterns, loops, and algorithmic thinking

Think of it like learning both poetry and mathematics: poetry teaches expressive language, mathematics teaches rigorous reasoning. Code brings the mathematical discipline to language modeling.

Code Corpus Sources

The primary source is GitHub, the world's largest collection of open-source code. Practitioners typically extract:

  • Public repositories across programming languages (Python, JavaScript, C++, etc.)
  • Documentation and README files (bridging code and natural language)
  • Code comments (explaining why, not just what)

Other sources include Stack Overflow discussions, technical documentation, and programming competition datasets.

Code-to-Text Ratios in Practice

Modern LLMs typically include 5-20% code in their pretraining mix. For example:

  • General-purpose models might use 8-15% code
  • Reasoning-focused models push toward 20%
  • Chat-optimized models might use less (5-10%)

This is carefully balanced with your Data Mix Ratios and Domain Balancing strategy—too much code can make the model less natural in conversational settings, too little sacrifices reasoning capabilities.

Key Takeaway: Including 5-20% code data in pretraining significantly enhances logical reasoning and compositional abilities, with GitHub serving as the primary corpus source for most modern LLMs.