Course contentsShow
← Machine Learning and Deep Learning
Lesson 3274 of 3,53871. Interpretability: Neural Network MethodsFree lesson

Induction Heads and In-Context Learning

Understanding the circuit mechanism enabling transformers to perform in-context learning through induction head patterns.

Induction Heads and In-Context Learning

What you'll learn: How a specific transformer circuit pattern called "induction heads" enables models to learn and repeat patterns from context without updating weights.

The Core Mechanism

Induction heads are a two-layer attention circuit discovered through mechanistic interpretability research. They explain how transformers perform in-context learning—the ability to recognize and complete patterns just from seeing examples in the prompt.

Here's how the circuit works:

  1. Previous token head (usually in an earlier layer): Looks back one token and copies information about what came after it previously
  2. Induction head (in a later layer): Attends to tokens that match the current context, then predicts what followed those tokens before

A Concrete Example

Imagine the input: "The cat sat. The dog ran. The cat ___"

  • When processing the final "cat", the induction head searches backward for previous instances of "cat"
  • It finds "cat" earlier and retrieves what followed it: "sat"
  • The model predicts "sat" should come next

This is pattern matching through attention composition—no gradient updates needed, just clever routing of information through attention layers.

Why This Matters

Induction heads are the first well-understood circuit that implements actual learning behavior (learning from context). They appear consistently across transformer architectures and explain capabilities like:

  • Few-shot learning from prompts
  • Translation after seeing examples
  • Completing repetitive patterns

This discovery validates the mechanistic interpretability approach: by analyzing activation patterns and using techniques like activation patching, researchers identified exactly which attention heads perform this function and how they compose across layers.

Key Takeaway: Induction heads are multi-layer attention circuits that enable transformers to perform in-context learning by finding previous pattern instances and copying their continuations—a mechanistic explanation of how prompting actually works.