Course contentsShow
← Security
Lesson 2818 of 3,05253. AI and ML SecurityFree lesson

Data Poisoning Attack Fundamentals

Understanding how attackers manipulate training data to compromise model behavior and integrity.

Data Poisoning Attack Fundamentals

What you'll learn: How attackers inject malicious data into training sets to corrupt machine learning models from the ground up.

The Core Threat

While adversarial examples attack deployed models at inference time, data poisoning targets the model during its most vulnerable phase: training. Attackers manipulate the training dataset itself—adding, modifying, or mislabeling samples—so the model learns incorrect patterns from the start.

Think of it like sabotaging a student's textbook before they study for an exam. No matter how smart the student is, if their study material is corrupted, they'll learn the wrong answers.

How Data Poisoning Works

Unlike evasion attacks (which you've seen with FGSM, PGD, and adversarial examples), poisoning happens before model deployment. The attacker needs access to influence the training pipeline:

  1. Inject malicious samples into the training set
  2. Mislabel existing data to confuse the model
  3. Bias the distribution to favor attacker-chosen behaviors

The poisoned model then makes systematic errors that benefit the attacker—whether that's misclassifying spam as legitimate, failing to detect malware, or creating backdoors triggered by specific inputs.

Attack Scenarios

Targeted poisoning: The attacker wants a specific input (like their malware signature) to be misclassified, while keeping overall accuracy high to avoid detection.

Availability attacks: Degrade the model's general performance by corrupting enough training data that it becomes unreliable.

Backdoor injection: Embed hidden triggers—the model performs normally except when it sees a secret pattern the attacker controls.

Why It Matters

Data poisoning is especially dangerous because it's stealthy and persistent. The corruption is baked into the model's weights, survives retraining if the poisoned data remains, and doesn't require runtime access to exploit.

Key Takeaway: Data poisoning attacks compromise ML models by corrupting training data, creating persistent vulnerabilities that traditional security measures won't detect since the model learned to be insecure.