Evaluating Defense Effectiveness
What you'll learn: How to properly assess whether ML defenses actually work using adaptive attacks, certified metrics, and by avoiding the gradient masking trap.
The Challenge of Defense Evaluation
Many proposed defenses against adversarial attacks look impressive on paper but crumble under scrutiny. The key problem? Gradient masking—when a defense obscures gradients without actually making the model more robust, fooling gradient-based attackers but not truly defending against adaptive adversaries.
Think of gradient masking like painting over rust on a car. It looks fixed, but the underlying weakness remains. An adaptive attacker—one who knows your defense and adjusts their strategy—will drive right through it.
Measuring Real Robustness
Adaptive Attacks
The gold standard for evaluation is testing against adaptive attacks: adversarial methods specifically tailored to your defense mechanism. If your defense uses input transformations, adaptive attacks should account for that transformation. If you ensemble models, attackers should target the ensemble directly, not individual models.
Simply showing that FGSM fails against your defense is insufficient—you must test whether sophisticated, defense-aware attackers can still succeed.
Certified Accuracy Metrics
Certified defenses provide provable robustness guarantees within a specified perturbation radius. When evaluating these, report:
- Certified accuracy at various ε (perturbation budgets)
- The radius within which predictions are provably unchanged
- Clean accuracy alongside certified accuracy
Unlike empirical defenses, certified methods give mathematical assurances that no attack within bounds can fool the model.
Avoiding Pitfalls
Red flags for gradient masking include: obfuscated gradients, defenses that only stop specific attacks, and suspiciously high robustness with minimal accuracy loss. Always test with multiple attack types and adaptive variants.
Key Takeaway: Defense effectiveness must be measured with adaptive attacks tailored to the defense mechanism, certified accuracy metrics when applicable, and careful checks for gradient masking rather than genuine robustness.