Activation Patching and Causal Interventions
What you'll learn: How to measure which activations in a neural network actually cause specific behaviors by surgically replacing them and watching what changes.
The Core Idea
Imagine you're debugging a machine, and you want to know which part is responsible for a specific output. You could replace one component with a known "clean" version and see if the behavior changes. Activation patching applies this same logic to neural networks.
Instead of just observing which neurons activate (correlation), you intervene by replacing activations from one forward pass with activations from a different context, then measure how the model's output changes. If swapping an activation dramatically alters behavior, you've found something causally important.
How It Works
- Run the model on an input where it produces behavior A (e.g., predicts "Paris")
- Run the model on a different input where it produces behavior B (e.g., predicts "London")
- Take activations from the second run and "patch" them into specific positions during the first run
- Measure how much the output shifts from A toward B
If patching a particular layer or neuron causes a large shift, that activation was causally responsible for the difference between behaviors A and B.
Why This Matters
Unlike attention visualization or gradient-based methods (which show correlation), activation patching reveals causal structure. It answers: "Does this activation make the model behave this way?" This is crucial for understanding circuits—the functional subgraphs that implement specific computations like detecting grammatical patterns or factual recall.
Key Takeaway: Activation patching surgically replaces internal activations to measure their causal impact on model behavior, moving beyond correlation to understand what truly drives neural network decisions.