Logit Attribution and Output Decomposition
What you'll learn: How to trace which components of a neural network contribute to final predictions by breaking down output logits into interpretable, additive pieces.
The Core Idea
Imagine you're debugging why your language model predicted "cat" instead of "dog." Rather than just seeing the final score, you'd like to know: which attention heads pushed toward "cat"? Which layers? Which input tokens?
Logit attribution decomposes the final output logit (the raw score before softmax) into a sum of contributions from individual network components. Because neural networks perform many linear transformations, we can often trace backwards and assign "credit" to specific parts—particular attention heads, layers, or even input tokens.
How It Works
The key insight: in transformer models, the residual stream acts as an accumulator. Each layer adds its contribution, and these additions are linear. At the end, the unembedding layer projects this accumulated representation to logits.
You can decompose a logit as:
logit[token] = sum of contributions from:
- Input embeddings
- Each attention head in each layer
- Each MLP block in each layer
- Positional encodings
This works because matrix multiplication distributes: (A + B + C) × W = A×W + B×W + C×W. You compute each component's effect separately and literally add them up to reconstruct the original logit.
Why This Matters
Unlike attention weights (which show where the model looks) or gradients (which show sensitivity), logit attribution reveals causal contributions: what actually pushed the prediction up or down. It's particularly powerful for understanding mechanistic circuits—you can identify which specific attention heads or layers are responsible for particular behaviors.
Key Takeaway: Logit attribution decomposes neural network outputs into additive contributions from individual components, revealing which parts of the model causally drive specific predictions through the residual stream.