Course contentsShow
← Machine Learning and Deep Learning
Lesson 2676 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Dynamic Sparse Training

Training sparse networks from scratch by periodically removing and regrowing connections during optimization.

Dynamic Sparse Training

What you'll learn: How to train sparse neural networks from scratch by periodically removing and regrowing connections during optimization, rather than pruning a dense network after training.

The Core Idea

Traditional pruning starts with a dense network, trains it fully, then removes weights. Dynamic Sparse Training (DST) flips this paradigm: you maintain a fixed sparsity level throughout training, periodically removing low-importance connections and regrowing new ones in promising locations.

Think of it like managing a forest: instead of growing all trees fully then cutting most down, you continuously remove weak saplings and plant new seeds in nutrient-rich soil, keeping a constant tree density.

How It Works

DST operates in cycles:

  1. Train for N steps with the current sparse mask
  2. Drop connections using magnitude-based pruning or gradient-based scoring (concepts from earlier lessons)
  3. Regrow connections where they're most needed—often where gradients are largest or randomly
  4. Repeat until training completes

The sparsity ratio stays constant (e.g., 90% sparse), but which connections exist changes over time. This allows the network to explore different sparse architectures and adapt its connectivity to the task.

Why It Matters

Unlike the Lottery Ticket Hypothesis (which requires finding the right sparse subnetwork), DST:

  • Trains sparse networks directly without needing a dense phase
  • Reduces memory and compute during training, not just inference
  • Often matches or exceeds the accuracy of pruned dense networks
  • Allows networks to "rewire" themselves as learning progresses

Key Takeaway: Dynamic Sparse Training maintains constant sparsity throughout optimization by periodically dropping weak connections and regrowing new ones, enabling efficient training of sparse networks from scratch without requiring an initial dense phase.