Course contentsShow
← Machine Learning and Deep Learning
Lesson 1635 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Classifier-Based Filtering

Training quality classifiers on curated data to filter web text and using perplexity-based filtering with existing language models.

Classifier-Based Filtering

What you'll learn: How to use trained classifiers to automatically select high-quality training data from massive web corpora.

The Core Idea

Imagine you're sorting through millions of web pages to build a training dataset. Rule-based filters (like checking character counts or punctuation ratios) catch obvious junk, but they miss subtle quality signals. Classifier-based filtering trains machine learning models to distinguish "good" from "bad" text, then uses these classifiers to score and filter your corpus.

How It Works

You start by curating a small, high-quality dataset—perhaps Wikipedia articles, books, or carefully selected web pages. This becomes your "positive" class. You contrast it with random web crawl data (your "negative" class). Train a classifier (often a simple model like logistic regression on text features, or a small neural network) to predict whether a document belongs to the high-quality set.

Once trained, run this classifier on your entire CommonCrawl corpus, scoring each document. Keep only documents above a confidence threshold—say, those the model is 80%+ confident resemble your curated data.

Perplexity-Based Filtering

An alternative approach uses an existing language model trained on high-quality data. Calculate the perplexity of each web document under this model. Low perplexity means the text resembles the model's training distribution (likely high-quality). High perplexity suggests unusual, potentially low-quality text. Filter by keeping documents below a perplexity threshold.

This leverages a pretrained model's learned understanding of "good" text without training a new classifier.

Key Takeaway: Classifier-based filtering uses machine learning to identify quality signals beyond simple rules, either through dedicated classifiers trained on curated examples or perplexity scores from existing language models trained on clean data.