Threat Modeling for Language Models
What You'll Learn
You'll learn how to systematically identify attack vectors, misuse scenarios, and failure modes in language models to make red teaming efforts more targeted and effective.
Why Threat Modeling Matters
Before you can effectively red team an LLM, you need a map of where things might go wrong. Threat modeling is the structured process of anticipating how your language model could be attacked, misused, or fail—before those problems emerge in production.
Think of it like a security audit for a building: you identify all the doors, windows, and weak points before someone tries to break in, not after.
Core Components of LLM Threat Modeling
1. Attack Vectors
Identify the technical pathways an adversary might exploit:
- Prompt injection points (user inputs, API calls, plugin interfaces)
- Training data poisoning opportunities
- Fine-tuning vulnerabilities
- API abuse patterns
2. Misuse Scenarios
Map out how legitimate features could be weaponized:
- Generating disinformation at scale
- Creating phishing content
- Automating social engineering
- Producing harmful instructions
- Bypassing content moderation
3. Failure Modes
Catalog ways the system might break down unintentionally:
- Hallucination in high-stakes domains (medical, legal)
- Bias amplification in decision-making contexts
- Privacy leakage through training data memorization
- Degraded performance under distribution shift
The Threat Modeling Process
Start by asking: Who might attack? Why? What capabilities do they have? Then map those to specific attack techniques (like the jailbreaks and adversarial methods you've learned) and potential defenses.
Key Takeaway: Effective threat modeling transforms vague security concerns into concrete attack scenarios and failure cases, enabling focused red teaming that actually finds vulnerabilities before adversaries do.