Course contentsShow
← Security
Lesson 2877 of 3,05253. AI and ML SecurityFree lesson

Malicious Pre-trained Models

Detecting backdoors in foundation models, risks of untrusted model weights, and safe model sourcing practices.

Malicious Pre-trained Models

You'll learn how attackers weaponize foundation models and how to safely source pre-trained weights.

The Hidden Threat in Model Weights

Pre-trained models are the backbone of modern AI development—but what if those "ready to use" weights contain hidden backdoors? Unlike traditional software where you inspect source code, neural network weights are opaque: billions of floating-point numbers with no human-readable logic. An attacker can embed malicious behavior during pre-training that activates only under specific conditions, and it's nearly impossible to spot by inspection alone.

How Backdoors Hide in Foundation Models

Think of a pre-trained model like a recipe from an untrusted chef. The ingredients (weights) look normal, but subtle poisoning during preparation means the dish becomes toxic only when served with a particular garnish (trigger). The backdoor was baked in during the original training—long before you fine-tuned it for your task.

Common risks:

  • Targeted misclassification: Model works perfectly except when specific triggers appear
  • Data exfiltration: Model subtly encodes sensitive information in outputs
  • Behavior hijacking: Fine-tuning doesn't remove the original malicious logic

Safe Model Sourcing Practices

  1. Verify provenance: Only download models from trusted, verified sources with reputation systems
  2. Check signatures: Use cryptographically signed model artifacts (building on lesson 2874's signing concepts)
  3. Behavioral testing: Run extensive test suites looking for anomalous outputs on edge cases
  4. Anomaly detection: Compare activation patterns against known-good baselines
  5. Staged deployment: Test heavily in sandboxed environments before production

Remember: model repositories can be compromised, and popular models can be trojanized. Always treat untrusted weights as potential malware.

Key Takeaway: Pre-trained model weights are an opaque attack vector—backdoors inserted during training persist through fine-tuning, making provenance verification and behavioral testing essential before deployment.