Course contentsShow
← Machine Learning and Deep Learning
Lesson 3275 of 3,53871. Interpretability: Neural Network MethodsFree lesson

Logit Attribution and Output Decomposition

Tracing how different components contribute to final predictions by decomposing output logits into additive contributions.

Logit Attribution and Output Decomposition

What you'll learn: How to trace which components of a neural network contribute to final predictions by breaking down output logits into interpretable, additive pieces.

The Core Idea

Imagine you're debugging why your language model predicted "cat" instead of "dog." Rather than just seeing the final score, you'd like to know: which attention heads pushed toward "cat"? Which layers? Which input tokens?

Logit attribution decomposes the final output logit (the raw score before softmax) into a sum of contributions from individual network components. Because neural networks perform many linear transformations, we can often trace backwards and assign "credit" to specific parts—particular attention heads, layers, or even input tokens.

How It Works

The key insight: in transformer models, the residual stream acts as an accumulator. Each layer adds its contribution, and these additions are linear. At the end, the unembedding layer projects this accumulated representation to logits.

You can decompose a logit as:

logit[token] = sum of contributions from:
  - Input embeddings
  - Each attention head in each layer
  - Each MLP block in each layer
  - Positional encodings

This works because matrix multiplication distributes: (A + B + C) × W = A×W + B×W + C×W. You compute each component's effect separately and literally add them up to reconstruct the original logit.

Why This Matters

Unlike attention weights (which show where the model looks) or gradients (which show sensitivity), logit attribution reveals causal contributions: what actually pushed the prediction up or down. It's particularly powerful for understanding mechanistic circuits—you can identify which specific attention heads or layers are responsible for particular behaviors.

Key Takeaway: Logit attribution decomposes neural network outputs into additive contributions from individual components, revealing which parts of the model causally drive specific predictions through the residual stream.