Pruning Schedules and Sparsity Targets
You'll learn how to control when and how much to prune during training to reach your desired sparsity level smoothly.
The Core Idea
When pruning a neural network, you rarely want to remove all weights at once—this would shock the model and destroy accuracy. Instead, you define a pruning schedule that gradually increases sparsity over time, and a sparsity target that specifies your final goal (e.g., 80% of weights pruned).
Think of it like losing weight: crash dieting (one-shot pruning to 80% sparsity) often fails, but gradually reducing calories over months (iterative pruning with a schedule) gives your body time to adapt.
Sparsity Targets
Your sparsity target is simply the final percentage of weights you want to prune. For example, a target of 0.75 means 75% of weights become zero at the end.
Common Pruning Schedules
Linear Ramp-Up
Sparsity increases linearly from 0% to your target over training steps. Simple but effective.
Example: If your target is 80% sparsity over 10,000 steps, at step 5,000 you're at 40% sparsity.
Polynomial Ramp-Up
Uses a polynomial curve (often cubic) to increase sparsity more gradually early on, then accelerate near the end. This gives the network more stable time to adjust initially.
Formula: s(t) = s_final * (1 - (1 - t/T)^n) where n controls the curve shape (typically 3).
The schedule determines when to prune (which training steps) and how much sparsity to enforce at each step, bridging the gap between your initialized dense model and your target sparse model.
Key Takeaway: Pruning schedules gradually ramp up sparsity during training to reach your target, with polynomial schedules offering smoother adaptation than linear ones.