Data Poisoning Attack Fundamentals
What you'll learn: How attackers inject malicious data into training sets to corrupt machine learning models from the ground up.
The Core Threat
While adversarial examples attack deployed models at inference time, data poisoning targets the model during its most vulnerable phase: training. Attackers manipulate the training dataset itself—adding, modifying, or mislabeling samples—so the model learns incorrect patterns from the start.
Think of it like sabotaging a student's textbook before they study for an exam. No matter how smart the student is, if their study material is corrupted, they'll learn the wrong answers.
How Data Poisoning Works
Unlike evasion attacks (which you've seen with FGSM, PGD, and adversarial examples), poisoning happens before model deployment. The attacker needs access to influence the training pipeline:
- Inject malicious samples into the training set
- Mislabel existing data to confuse the model
- Bias the distribution to favor attacker-chosen behaviors
The poisoned model then makes systematic errors that benefit the attacker—whether that's misclassifying spam as legitimate, failing to detect malware, or creating backdoors triggered by specific inputs.
Attack Scenarios
Targeted poisoning: The attacker wants a specific input (like their malware signature) to be misclassified, while keeping overall accuracy high to avoid detection.
Availability attacks: Degrade the model's general performance by corrupting enough training data that it becomes unreliable.
Backdoor injection: Embed hidden triggers—the model performs normally except when it sees a secret pattern the attacker controls.
Why It Matters
Data poisoning is especially dangerous because it's stealthy and persistent. The corruption is baked into the model's weights, survives retraining if the poisoned data remains, and doesn't require runtime access to exploit.
Key Takeaway: Data poisoning attacks compromise ML models by corrupting training data, creating persistent vulnerabilities that traditional security measures won't detect since the model learned to be insecure.