Course contentsShow
← Machine Learning and Deep Learning
Lesson 2678 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Implementing Pruning in PyTorch

Using torch.nn.utils.prune for magnitude pruning, creating custom pruning methods, and managing sparse masks.

Implementing Pruning in PyTorch

You'll learn how to use PyTorch's built-in pruning utilities to remove unimportant weights, create custom pruning strategies, and manage the resulting sparse parameter masks.

The Core Idea

PyTorch provides torch.nn.utils.prune as a toolkit for systematically removing weights from neural networks. This module applies pruning masks to parameters—essentially creating zero-valued weights without physically deleting them. Think of it like placing transparent sticky notes over unwanted text in a book: the words remain, but they're effectively invisible and ignored during computation.

Using torch.nn.utils.prune for Magnitude Pruning

The simplest approach is magnitude pruning, which zeros out the smallest weights:

import torch
import torch.nn.utils.prune as prune

# Prune 30% of weights with smallest magnitude
model = MyNeuralNetwork()
prune.l1_unstructured(model.conv1, name='weight', amount=0.3)

This creates two things: a weight_orig parameter (original values) and a weight_mask (binary mask). The actual weight becomes a computed property: weight_orig * weight_mask.

Creating Custom Pruning Methods

You can subclass BasePruningMethod to define custom importance criteria:

class ThresholdPruning(prune.BasePruningMethod):
    PRUNING_TYPE = 'unstructured'
    
    def compute_mask(self, tensor, default_mask):
        return torch.abs(tensor) > 0.01  # Keep weights > threshold

Managing Sparse Masks

Masks accumulate across multiple pruning operations. To make pruning permanent (remove mask overhead):

prune.remove(model.conv1, 'weight')  # Applies mask permanently

Key Takeaway: PyTorch's pruning API uses parameter masks to zero weights non-destructively, supports both built-in magnitude pruning and custom strategies, and requires explicit removal to finalize the sparse structure.