Implementing Pruning in PyTorch
You'll learn how to use PyTorch's built-in pruning utilities to remove unimportant weights, create custom pruning strategies, and manage the resulting sparse parameter masks.
The Core Idea
PyTorch provides torch.nn.utils.prune as a toolkit for systematically removing weights from neural networks. This module applies pruning masks to parameters—essentially creating zero-valued weights without physically deleting them. Think of it like placing transparent sticky notes over unwanted text in a book: the words remain, but they're effectively invisible and ignored during computation.
Using torch.nn.utils.prune for Magnitude Pruning
The simplest approach is magnitude pruning, which zeros out the smallest weights:
import torch
import torch.nn.utils.prune as prune
# Prune 30% of weights with smallest magnitude
model = MyNeuralNetwork()
prune.l1_unstructured(model.conv1, name='weight', amount=0.3)
This creates two things: a weight_orig parameter (original values) and a weight_mask (binary mask). The actual weight becomes a computed property: weight_orig * weight_mask.
Creating Custom Pruning Methods
You can subclass BasePruningMethod to define custom importance criteria:
class ThresholdPruning(prune.BasePruningMethod):
PRUNING_TYPE = 'unstructured'
def compute_mask(self, tensor, default_mask):
return torch.abs(tensor) > 0.01 # Keep weights > threshold
Managing Sparse Masks
Masks accumulate across multiple pruning operations. To make pruning permanent (remove mask overhead):
prune.remove(model.conv1, 'weight') # Applies mask permanently
Key Takeaway: PyTorch's pruning API uses parameter masks to zero weights non-destructively, supports both built-in magnitude pruning and custom strategies, and requires explicit removal to finalize the sparse structure.