Fast Adversarial Training
What you'll learn: How to dramatically speed up adversarial training using FGSM while avoiding a critical failure mode called catastrophic overfitting.
The Speed Problem
Standard adversarial training using PGD (which you've already learned) is extremely slow. Each training example requires multiple gradient computations to generate strong adversarial examples—typically 7-10 steps. This makes training 5-10× slower than normal.
The Fast Solution: FGSM-Based Training
Fast Adversarial Training replaces multi-step PGD attacks with single-step FGSM during training. Since FGSM requires only one gradient computation per example, training time drops dramatically—approaching the speed of standard training.
The appeal is obvious: why spend days training when you could finish in hours?
The Catastrophic Overfitting Trap
Here's the catch: models trained purely with FGSM often exhibit catastrophic overfitting. The model appears robust against FGSM attacks during training, achieving high robust accuracy. But when evaluated against stronger multi-step attacks like PGD, it completely fails—sometimes worse than an undefended model.
Think of it like learning to defend against one specific boxing combo. You get great at blocking that exact sequence, but a slightly different attack catches you completely off-guard.
This happens because FGSM produces weaker, less diverse adversarial examples. The model "overfits" to FGSM's characteristic perturbation patterns without learning genuine robustness.
Making Fast Training Work
Recent methods combine FGSM with regularization techniques (random initialization of perturbations, early stopping, specific learning rate schedules) to prevent catastrophic overfitting while maintaining speed advantages. The key is introducing enough diversity in the single-step attacks.
Key Takeaway: Fast adversarial training uses FGSM to dramatically reduce training time, but naive implementation leads to catastrophic overfitting where models fail against stronger attacks—successful approaches require careful regularization to maintain genuine robustness.