Course contentsShow
← Machine Learning and Deep Learning
Lesson 1628 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Emergent Abilities and Phase Transitions

How certain capabilities appear suddenly at specific scale thresholds rather than improving gradually.

Emergent Abilities and Phase Transitions

What you'll learn: How certain capabilities in large language models appear suddenly at specific scale thresholds rather than improving gradually.

The Phenomenon

As we've explored scaling laws, we've seen mostly smooth, predictable improvements: double the parameters, get X% better performance. But something surprising happens at certain scales—capabilities that were completely absent suddenly "turn on" like a light switch.

These are emergent abilities: skills that only appear once a model crosses a specific size threshold. Below that threshold, the model performs at random-chance level. Above it, performance jumps dramatically.

Examples of Emergent Behaviors

Common emergent abilities include:

  • Few-shot arithmetic: Models below ~10B parameters can't do 3-digit addition reliably; larger models can
  • Multi-step reasoning: Chain-of-thought reasoning emerges around 60-100B parameters
  • Following complex instructions: Understanding multi-part prompts requires sufficient scale
  • In-context learning: The ability to learn new tasks from examples in the prompt itself

Why Phase Transitions Occur

Think of water freezing: at 0°C, liquid suddenly becomes solid—there's no "partially frozen" middle state at the molecular level (though there appears to be at bulk scale). Similarly, certain neural representations or computational patterns may require a minimum network capacity to form at all.

The working theory: some capabilities require coordinated representations across many layers and attention heads. Below a threshold size, the model simply cannot maintain the necessary internal structure.

Key Takeaway: Emergent abilities reveal that scaling isn't just about incremental improvements—certain critical capabilities only unlock at specific model sizes, creating phase transitions where models suddenly gain qualitatively new skills rather than just quantitatively better performance.