Course contentsShow
← Machine Learning and Deep Learning
Lesson 1639 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Handling Personally Identifiable Information

Detecting and removing PII from training data including names, addresses, and sensitive content for privacy and safety.

Handling Personally Identifiable Information

You'll learn how to detect and remove sensitive personal data from training corpora to protect privacy and ensure safety.

What Is PII and Why Does It Matter?

When you scrape billions of web pages for pretraining data (like CommonCrawl), you inevitably collect Personally Identifiable Information (PII)—names, email addresses, phone numbers, physical addresses, social security numbers, medical records, and other sensitive content. Training a large language model on this data creates two serious problems:

  1. Privacy violation: The model might memorize and later reproduce someone's private information
  2. Safety risk: The model could leak sensitive data during inference, potentially causing real harm

Think of PII like poison in a water supply—even small amounts contaminate the whole system. Unlike quality filtering or deduplication (which improve model performance), PII removal is about ethical responsibility.

Detection and Removal Strategies

Pattern-based detection uses regular expressions to find structured PII like email formats (\S+@\S+\.\S+) or phone numbers. This catches obvious cases but misses context-dependent information.

Named Entity Recognition (NER) models identify person names, locations, and organizations in context. You might flag documents containing too many detected entities.

Classifier-based filtering (mentioned in your earlier lessons) can train models specifically to detect PII-heavy documents, then exclude them entirely or redact specific spans.

The challenge: being aggressive enough to protect privacy without removing too much valuable training data. Some organizations replace detected PII with placeholders ([NAME], [EMAIL]) rather than deleting entire documents.

Key Takeaway: Detecting and removing PII from pretraining data is a critical ethical step that uses pattern matching, NER models, and classifiers to prevent large language models from memorizing and reproducing sensitive personal information.