Course contentsShow
← Machine Learning and Deep Learning
Lesson 1633 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Quality Filtering: Heuristics and Rules

Basic filtering techniques including language detection, length filters, profanity removal, and removing low-quality or duplicate content.

Quality Filtering: Heuristics and Rules

What you'll learn: How to clean massive web-crawled datasets using simple but effective filtering rules before training large language models.

Why Quality Filtering Matters

When you harvest billions of web pages from CommonCrawl, you get everything—the good, the bad, and the gibberish. Training a language model on raw web data is like trying to learn cooking from every random note ever written in a kitchen: you'd learn recipes alongside grocery lists, rants, and coffee stains.

Quality filtering applies heuristic rules to automatically remove low-quality content before pretraining begins.

Core Filtering Techniques

Language Detection

Remove documents not in your target language(s). Libraries like langdetect or fastText classify text by language. If you're building an English model, filter out non-English pages.

Length Filters

  • Too short: Single-sentence pages or stub articles contain little useful information
  • Too long: Extremely long documents may be corrupted data dumps or spam
  • Typical threshold: keep documents between 100–100,000 tokens

Profanity and Toxicity Removal

Use blocklists or classifiers to remove explicit content, hate speech, or toxic text. This reduces harmful outputs during generation.

Duplicate Detection

Web crawls contain many duplicates. Remove exact or near-duplicate documents using:

  • Exact hashing (MD5/SHA)
  • MinHash/LSH for near-duplicates
  • n-gram overlap detection

Quality Heuristics

Flag and remove pages with:

  • Too many uppercase words (likely spam)
  • Excessive punctuation or special characters
  • Low word-to-symbol ratios
  • Repeated lines or boilerplate patterns
  • HTML artifacts (<div>, JavaScript fragments)

The Trade-off

Aggressive filtering improves average quality but risks removing valid edge-case content. Conservative filtering preserves diversity but includes noise. Modern LLM pipelines typically filter 30-60% of raw web data.

Key Takeaway: Quality filtering uses simple, rule-based heuristics (language, length, profanity, duplicates, formatting patterns) to remove low-quality web content, dramatically improving pretraining data quality at the cost of some dataset diversity.