One-Shot vs Iterative Pruning
What you'll learn: How to decide between pruning your neural network all at once versus gradually removing weights over multiple rounds with retraining in between.
The Core Idea
Once you've decided to prune your network using magnitude-based pruning (removing the smallest weights), you face a critical strategy decision: should you remove all target weights immediately, or gradually chip away at them over time?
One-shot pruning means you identify and remove all weights below your threshold in a single pass. For example, if you want to prune 80% of weights, you do it all at once, then retrain the remaining network.
Iterative pruning takes a gradual approach: prune a smaller percentage (say 20%), retrain the network to recover accuracy, then prune another 20%, retrain again, and repeat until you reach your target sparsity level.
Why It Matters
Think of pruning like trimming a tree. One-shot pruning is like cutting all the branches you plan to remove in one day—fast but shocking to the tree. Iterative pruning is like trimming a little each week, giving the tree time to adapt and redirect nutrients.
One-shot is computationally cheaper (only one retraining cycle) but often leads to larger accuracy drops, especially at high sparsity levels. The network gets a massive shock and may struggle to recover.
Iterative pruning is more forgiving. The network adjusts gradually, learning to compensate for lost weights at each stage. This typically preserves accuracy better but requires multiple training cycles, making it more expensive.
Key Takeaway: One-shot pruning is faster but riskier for accuracy; iterative pruning is slower but typically maintains better performance by allowing the network to adapt gradually to increasing sparsity.