Gradient-Based Importance Scoring
What you'll learn: How to use gradient information to determine which weights contribute most to model performance before pruning.
The Core Idea
When deciding which weights to prune, we can look beyond simple magnitude (absolute value) and instead ask: "How much does changing this weight affect the loss?" Gradient-based importance scoring uses first-order gradients (∂L/∂w) or second-order information (curvature) to identify which parameters matter most to your model's predictions.
Think of it like this: magnitude tells you how big a weight is, but gradients tell you how sensitive your model's output is to that weight. A large weight with a tiny gradient might be safely pruned, while a small weight with a large gradient could be critical.
First-Order vs Second-Order Approaches
First-order methods use the gradient ∂L/∂w directly. A common importance score is simply |∂L/∂w|—weights with large gradients are changing the loss significantly and should be kept.
Second-order methods consider the Hessian (∂²L/∂w²), which captures how the gradient itself changes. This reveals curvature: a weight in a flat region (low curvature) can be pruned with minimal impact, while weights in sharp valleys (high curvature) are crucial. The challenge? Computing the full Hessian is expensive, so diagonal approximations are often used.
Why It Matters
Gradient-based scoring often outperforms pure magnitude pruning because it incorporates how the model uses each weight during training, not just the weight's scale. This leads to smarter pruning decisions, especially when combined with iterative pruning and fine-tuning strategies you've already learned.
Key Takeaway: Gradient-based importance scoring leverages first or second-order derivative information to identify which weights most affect model performance, enabling more informed pruning decisions than magnitude alone.