Course contentsShow
← Machine Learning and Deep Learning
Lesson 3448 of 3,53875. LLM Safety and Alignment ChallengesFree lesson

Threat Modeling for Language Models

Identifying potential attack vectors, misuse scenarios, and failure modes to guide red teaming efforts.

Threat Modeling for Language Models

What You'll Learn

You'll learn how to systematically identify attack vectors, misuse scenarios, and failure modes in language models to make red teaming efforts more targeted and effective.

Why Threat Modeling Matters

Before you can effectively red team an LLM, you need a map of where things might go wrong. Threat modeling is the structured process of anticipating how your language model could be attacked, misused, or fail—before those problems emerge in production.

Think of it like a security audit for a building: you identify all the doors, windows, and weak points before someone tries to break in, not after.

Core Components of LLM Threat Modeling

1. Attack Vectors

Identify the technical pathways an adversary might exploit:

  • Prompt injection points (user inputs, API calls, plugin interfaces)
  • Training data poisoning opportunities
  • Fine-tuning vulnerabilities
  • API abuse patterns

2. Misuse Scenarios

Map out how legitimate features could be weaponized:

  • Generating disinformation at scale
  • Creating phishing content
  • Automating social engineering
  • Producing harmful instructions
  • Bypassing content moderation

3. Failure Modes

Catalog ways the system might break down unintentionally:

  • Hallucination in high-stakes domains (medical, legal)
  • Bias amplification in decision-making contexts
  • Privacy leakage through training data memorization
  • Degraded performance under distribution shift

The Threat Modeling Process

Start by asking: Who might attack? Why? What capabilities do they have? Then map those to specific attack techniques (like the jailbreaks and adversarial methods you've learned) and potential defenses.

Key Takeaway: Effective threat modeling transforms vague security concerns into concrete attack scenarios and failure cases, enabling focused red teaming that actually finds vulnerabilities before adversaries do.