Quality Filtering: Heuristics and Rules
What you'll learn: How to clean massive web-crawled datasets using simple but effective filtering rules before training large language models.
Why Quality Filtering Matters
When you harvest billions of web pages from CommonCrawl, you get everything—the good, the bad, and the gibberish. Training a language model on raw web data is like trying to learn cooking from every random note ever written in a kitchen: you'd learn recipes alongside grocery lists, rants, and coffee stains.
Quality filtering applies heuristic rules to automatically remove low-quality content before pretraining begins.
Core Filtering Techniques
Language Detection
Remove documents not in your target language(s). Libraries like langdetect or fastText classify text by language. If you're building an English model, filter out non-English pages.
Length Filters
- Too short: Single-sentence pages or stub articles contain little useful information
- Too long: Extremely long documents may be corrupted data dumps or spam
- Typical threshold: keep documents between 100–100,000 tokens
Profanity and Toxicity Removal
Use blocklists or classifiers to remove explicit content, hate speech, or toxic text. This reduces harmful outputs during generation.
Duplicate Detection
Web crawls contain many duplicates. Remove exact or near-duplicate documents using:
- Exact hashing (MD5/SHA)
- MinHash/LSH for near-duplicates
- n-gram overlap detection
Quality Heuristics
Flag and remove pages with:
- Too many uppercase words (likely spam)
- Excessive punctuation or special characters
- Low word-to-symbol ratios
- Repeated lines or boilerplate patterns
- HTML artifacts (
<div>, JavaScript fragments)
The Trade-off
Aggressive filtering improves average quality but risks removing valid edge-case content. Conservative filtering preserves diversity but includes noise. Modern LLM pipelines typically filter 30-60% of raw web data.
Key Takeaway: Quality filtering uses simple, rule-based heuristics (language, length, profanity, duplicates, formatting patterns) to remove low-quality web content, dramatically improving pretraining data quality at the cost of some dataset diversity.