Studying Emergent Algorithms in Language Models
What you'll learn: How researchers reverse-engineer specific computational strategies that language models spontaneously develop during training.
The Core Idea
Through mechanistic interpretability techniques, researchers have discovered that language models don't just memorize patterns—they develop discrete, human-understandable algorithms to solve tasks. These emergent algorithms are computational strategies that arise naturally from training, not from explicit programming.
Case Studies of Discovered Circuits
Indirect Object Identification (IOI)
In sentences like "When John and Mary went to the store, John gave a drink to...", models learn a multi-step algorithm to predict "Mary." The circuit involves:
- Name mover heads that copy the indirect object token
- Duplicate token heads that detect which name appears twice (John)
- Inhibition mechanisms that suppress the repeated name
- S-inhibition heads that handle the subject position
This isn't random pattern matching—it's a structured algorithm with clear computational roles.
Greater-Than Comparison
Models learn to compare numbers by decomposing the task:
- Early layers encode magnitude information
- Middle layers perform digit-by-digit comparison from left to right
- Late layers aggregate results into a final decision
The model essentially rediscovers a place-value algorithm humans use.
Why This Matters
These discoveries show that neural networks can be understood as algorithmic systems, not just black boxes. By using activation patching, logit attribution, and attention head analysis, we can map how information flows through specific pathways to implement these strategies.
Key Takeaway: Language models spontaneously develop structured, interpretable algorithms like indirect object identification and numerical comparison—circuits we can reverse-engineer using mechanistic interpretability tools to understand not just what models learn, but how they compute.