Course contentsShow
← Machine Learning and Deep Learning
Lesson 3330 of 3,53873. Privacy in Machine LearningFree lesson

Training Data Extraction from Language Models

Explore how large language models can memorize and regurgitate verbatim training data through carefully crafted prompts.

Training Data Extraction from Language Models

What you'll learn: How attackers can prompt large language models to spit out exact quotes from their training data, exposing private information.

The Core Problem

Large language models like GPT-3, BERT, and their descendants are trained on massive text datasets scraped from the internet. While these models learn general patterns, they sometimes memorize specific sequences verbatim—especially when that data appears multiple times or is particularly unusual. Through cleverly crafted prompts, an attacker can trick a model into regurgitating these memorized snippets.

Think of it like this: imagine you studied for an exam by reading thousands of books. Most of the time you'd answer questions using general knowledge. But occasionally, a particularly striking quote might be burned into your memory word-for-word. If someone asked the right question, you'd recite it perfectly—perhaps without realizing you're exposing the exact source material.

How the Attack Works

Researchers discovered that prompting a language model with a partial phrase (like the beginning of an email, a credit card number prefix, or a person's name) can cause it to "complete" the sequence with exact training data. This is particularly dangerous when training sets accidentally include:

  • Personal emails or messages
  • API keys or passwords
  • Private medical records
  • Confidential business documents
  • Copyrighted text

The attack doesn't require access to model weights—just the ability to query the model repeatedly with different prompts and observe which completions look suspiciously specific or formatted (like phone numbers, addresses, or code).

Why It Matters

Unlike membership inference (which tells you if data was in training) or model inversion (which reconstructs approximate features), extraction attacks recover exact, usable training examples. This represents a severe privacy breach when sensitive information was inadvertently included during training.

Key Takeaway: Large language models can memorize and leak verbatim training data through carefully designed prompts, making data curation and filtering critical privacy safeguards before training begins.