Course contentsShow
← Machine Learning and Deep Learning
Lesson 2668 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Magnitude-Based Pruning Fundamentals

Learning the simplest pruning criterion: removing weights with smallest absolute values based on importance assumptions.

Magnitude-Based Pruning Fundamentals

What you'll learn: How to identify and remove the least important weights in a neural network using their absolute values as a simple importance criterion.

The Core Idea

Magnitude-based pruning is the simplest and most intuitive pruning method: remove weights with the smallest absolute values. The underlying assumption is straightforward—if a weight is close to zero, it contributes little to the network's output, so we can safely set it to exactly zero without significantly harming performance.

Think of it like pruning a tree: you remove the smallest, weakest branches first because they contribute least to the tree's overall structure and health. Similarly, small weights have minimal impact on your model's predictions.

How It Works

The process is remarkably simple:

  1. Calculate absolute values of all weights (or weights in a specific layer)
  2. Rank them from smallest to largest
  3. Choose a pruning ratio (e.g., remove 50% of weights)
  4. Set the smallest weights to zero according to your ratio

For unstructured pruning, you evaluate every individual weight. For structured pruning, you might sum magnitudes across entire channels or filters before deciding what to remove.

Why Magnitude?

This criterion relies on a key assumption: magnitude correlates with importance. A weight of 0.001 likely matters less than one of 3.7. While not perfect (context matters!), magnitude pruning is computationally cheap, requires no retraining to identify candidates, and works surprisingly well in practice as a baseline method.

Key Takeaway: Magnitude-based pruning removes weights with the smallest absolute values, operating on the simple but effective assumption that small weights contribute least to model predictions.