Course contentsShow
← Machine Learning and Deep Learning
Lesson 3387 of 3,53874. Adversarial RobustnessFree lesson

Threat Models and Attack Scenarios

Define white-box, black-box, and gray-box attack scenarios and understand the attacker's knowledge and capabilities.

Threat Models and Attack Scenarios

What you'll learn: How to categorize adversarial attacks based on what the attacker knows about your model and what they can access.

Understanding Attacker Knowledge Levels

In adversarial machine learning, we classify attacks into three threat models based on the attacker's knowledge and capabilities:

White-Box Attacks

The attacker has complete knowledge of your model: architecture, parameters, training data, and gradients. They can peer inside the "white box" and exploit this information to craft optimal adversarial perturbations. This is the strongest threat model—imagine an insider with full system access designing the perfect attack.

Black-Box Attacks

The attacker treats your model as a closed black box. They can only query it with inputs and observe outputs (predictions, confidence scores). They don't know the architecture or weights. Think of this like a customer using an API—they can test inputs but can't see the internals. Attackers often use techniques like transferability or query-based optimization.

Gray-Box Attacks

The attacker has partial knowledge—perhaps they know the model architecture but not the exact trained weights, or they have access to similar training data. This represents a middle ground: more realistic than white-box but more dangerous than pure black-box scenarios.

Why This Matters

Understanding these threat models helps you:

  • Assess realistic risks for your deployment scenario (is your model exposed via API? Is the code open-source?)
  • Prioritize defenses based on likely attack vectors
  • Evaluate robustness claims in research papers (white-box robustness is harder to achieve)

Most real-world scenarios fall between black-box and gray-box, since attackers rarely have perfect knowledge but often know something about your system.

Key Takeaway: Threat models define what an attacker knows and can do—white-box means full access, black-box means query-only access, and gray-box sits in between. Your defense strategy should match your deployment's threat model.