Course contentsShow
← Machine Learning and Deep Learning
Lesson 2670 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Pruning Schedules and Sparsity Targets

Designing when and how much to prune during training, including linear and polynomial sparsity ramp-ups.

Pruning Schedules and Sparsity Targets

You'll learn how to control when and how much to prune during training to reach your desired sparsity level smoothly.

The Core Idea

When pruning a neural network, you rarely want to remove all weights at once—this would shock the model and destroy accuracy. Instead, you define a pruning schedule that gradually increases sparsity over time, and a sparsity target that specifies your final goal (e.g., 80% of weights pruned).

Think of it like losing weight: crash dieting (one-shot pruning to 80% sparsity) often fails, but gradually reducing calories over months (iterative pruning with a schedule) gives your body time to adapt.

Sparsity Targets

Your sparsity target is simply the final percentage of weights you want to prune. For example, a target of 0.75 means 75% of weights become zero at the end.

Common Pruning Schedules

Linear Ramp-Up

Sparsity increases linearly from 0% to your target over training steps. Simple but effective.

Example: If your target is 80% sparsity over 10,000 steps, at step 5,000 you're at 40% sparsity.

Polynomial Ramp-Up

Uses a polynomial curve (often cubic) to increase sparsity more gradually early on, then accelerate near the end. This gives the network more stable time to adjust initially.

Formula: s(t) = s_final * (1 - (1 - t/T)^n) where n controls the curve shape (typically 3).

The schedule determines when to prune (which training steps) and how much sparsity to enforce at each step, bridging the gap between your initialized dense model and your target sparse model.

Key Takeaway: Pruning schedules gradually ramp up sparsity during training to reach your target, with polynomial schedules offering smoother adaptation than linear ones.