Induction Heads and In-Context Learning
What you'll learn: How a specific transformer circuit pattern called "induction heads" enables models to learn and repeat patterns from context without updating weights.
The Core Mechanism
Induction heads are a two-layer attention circuit discovered through mechanistic interpretability research. They explain how transformers perform in-context learning—the ability to recognize and complete patterns just from seeing examples in the prompt.
Here's how the circuit works:
- Previous token head (usually in an earlier layer): Looks back one token and copies information about what came after it previously
- Induction head (in a later layer): Attends to tokens that match the current context, then predicts what followed those tokens before
A Concrete Example
Imagine the input: "The cat sat. The dog ran. The cat ___"
- When processing the final "cat", the induction head searches backward for previous instances of "cat"
- It finds "cat" earlier and retrieves what followed it: "sat"
- The model predicts "sat" should come next
This is pattern matching through attention composition—no gradient updates needed, just clever routing of information through attention layers.
Why This Matters
Induction heads are the first well-understood circuit that implements actual learning behavior (learning from context). They appear consistently across transformer architectures and explain capabilities like:
- Few-shot learning from prompts
- Translation after seeing examples
- Completing repetitive patterns
This discovery validates the mechanistic interpretability approach: by analyzing activation patterns and using techniques like activation patching, researchers identified exactly which attention heads perform this function and how they compose across layers.
Key Takeaway: Induction heads are multi-layer attention circuits that enable transformers to perform in-context learning by finding previous pattern instances and copying their continuations—a mechanistic explanation of how prompting actually works.