Course contentsShow
← Machine Learning and Deep Learning
Lesson 3406 of 3,53874. Adversarial RobustnessFree lesson

Adversarial Training Trade-offs

Understanding the accuracy-robustness trade-off and computational costs of adversarial training.

Adversarial Training Trade-offs

You'll learn: Why adversarial training improves robustness but comes with significant costs in accuracy and computation.

The Core Problem

Adversarial training (lessons 3403-3405) makes models resistant to adversarial attacks by including perturbed examples during training. However, this defense doesn't come free—it introduces fundamental trade-offs that every practitioner must navigate.

The Accuracy-Robustness Trade-off

When you train a model to be robust against adversarial perturbations, you're essentially asking it to ignore small variations in input that might be semantically meaningless. This makes the decision boundary "smoother" but also more conservative.

The result: Robust models typically suffer 5-15% accuracy drops on clean (unperturbed) data compared to standard training. It's like training a security guard to be extremely cautious—they'll catch more threats but also question legitimate visitors more often.

This isn't just an implementation detail; research suggests it may be a fundamental trade-off for complex datasets. The model capacity needed to achieve both high clean accuracy and robustness simultaneously exceeds what standard architectures can provide.

Computational Costs

Adversarial training is computationally expensive:

  • Training time: 2-10× longer than standard training (you must generate adversarial examples on-the-fly using methods like PGD)
  • Memory usage: Higher due to storing perturbations and extra forward/backward passes
  • Hyperparameter sensitivity: Requires careful tuning of perturbation budgets, step sizes, and iteration counts

Fast adversarial training (lesson 3405) helps but still costs more than standard training.

Key Takeaway: Adversarial training forces you to balance three competing goals—clean accuracy, adversarial robustness, and computational efficiency—with no free lunch available; practitioners must carefully weigh these trade-offs based on their application's threat model and resource constraints.