Threat Model for Adversarial Attacks
What You'll Learn
You'll learn how security researchers categorize attacker capabilities when analyzing adversarial machine learning attacks—defining what information and access levels adversaries possess.
Understanding the Threat Landscape
In adversarial machine learning, the threat model defines the rules of engagement: what can an attacker see, touch, and manipulate? Just like physical security distinguishes between an outsider picking a lock versus an insider with building keys, we must classify adversarial capabilities.
White-Box vs Black-Box Access
White-box access means the attacker has complete knowledge: the model architecture, all trained weights, training data, and internal operations. Think of it as having the source code and blueprints. This represents the worst-case scenario but enables researchers to find vulnerabilities systematically.
Black-box access means the attacker only interacts with the model as a user would—submitting inputs and observing outputs. No internal knowledge. Like testing a locked vault by trying combinations without seeing the mechanism inside.
Gray-box sits between these extremes: perhaps the attacker knows the architecture but not the trained weights, or has partial information.
Query Budgets
Real-world attackers face constraints. A query budget limits how many times an attacker can probe the model. Cloud-based ML APIs often charge per query or implement rate limiting—attackers must succeed with perhaps 1,000 queries instead of millions.
Perturbation Constraints
How much can the attacker modify inputs? A spam filter adversary must keep emails readable to humans. An autonomous vehicle attacker might only add small stickers to stop signs. These perturbation constraints define what modifications are "legal" within the attack scenario—often measured by mathematical distance metrics or perceptual similarity bounds.
Key Takeaway:
Threat models for adversarial attacks formalize attacker capabilities through access level (white/black/gray-box), query budgets, and perturbation constraints—establishing realistic boundaries for security analysis.