Emergent Abilities and Phase Transitions
What you'll learn: How certain capabilities in large language models appear suddenly at specific scale thresholds rather than improving gradually.
The Phenomenon
As we've explored scaling laws, we've seen mostly smooth, predictable improvements: double the parameters, get X% better performance. But something surprising happens at certain scales—capabilities that were completely absent suddenly "turn on" like a light switch.
These are emergent abilities: skills that only appear once a model crosses a specific size threshold. Below that threshold, the model performs at random-chance level. Above it, performance jumps dramatically.
Examples of Emergent Behaviors
Common emergent abilities include:
- Few-shot arithmetic: Models below ~10B parameters can't do 3-digit addition reliably; larger models can
- Multi-step reasoning: Chain-of-thought reasoning emerges around 60-100B parameters
- Following complex instructions: Understanding multi-part prompts requires sufficient scale
- In-context learning: The ability to learn new tasks from examples in the prompt itself
Why Phase Transitions Occur
Think of water freezing: at 0°C, liquid suddenly becomes solid—there's no "partially frozen" middle state at the molecular level (though there appears to be at bulk scale). Similarly, certain neural representations or computational patterns may require a minimum network capacity to form at all.
The working theory: some capabilities require coordinated representations across many layers and attention heads. Below a threshold size, the model simply cannot maintain the necessary internal structure.
Key Takeaway: Emergent abilities reveal that scaling isn't just about incremental improvements—certain critical capabilities only unlock at specific model sizes, creating phase transitions where models suddenly gain qualitatively new skills rather than just quantitatively better performance.