Linear Warmup
What You'll Learn
You'll learn how gradually increasing the learning rate from near-zero to its target value during the first few training steps can prevent unstable, erratic optimization at the start of training.
The Core Idea
When training begins, your model's weights are randomly initialized and far from optimal. If you immediately apply a large learning rate, the optimizer might make wild, destabilizing updates that send training off course or cause loss to explode.
Linear warmup solves this by starting with a very small learning rate (often close to zero) and gradually increasing it linearly over a fixed number of steps or epochs until it reaches your desired target learning rate. Think of it like gradually accelerating a car from rest rather than flooring the gas pedal immediately—you gain control and stability.
Why It Matters
Early in training, gradients can be noisy and unpredictable. A small initial learning rate lets the model "find its footing," making gentle updates while the network stabilizes. As weights begin to improve, the learning rate ramps up to enable faster convergence.
This technique is especially valuable when:
- Using adaptive optimizers like Adam with large batch sizes
- Training very deep networks
- Working with transformer architectures
How It Works
During warmup (e.g., first 1000-5000 steps):
current_lr = initial_lr + (target_lr - initial_lr) × (step / warmup_steps)
After warmup completes, you typically switch to another schedule like cosine annealing or step decay.
Key Takeaway: Linear warmup stabilizes early training by gradually ramping up the learning rate from near-zero to target, preventing erratic updates when gradients are most unpredictable.