BERT Pretraining Data: BookCorpus and Wikipedia
What You'll Learn
You'll understand the massive text corpora used to pretrain BERT and why their size and diversity are crucial for learning rich language representations.
The Foundation: Where BERT Learns Language
BERT wasn't trained on random internet text—its creators carefully selected two large-scale datasets that together provided over 3 billion words of high-quality English text:
BookCorpus
The BookCorpus dataset contains over 11,000 unpublished books spanning diverse genres: romance, fantasy, adventure, science fiction, and more. This corpus provides:
- Natural, flowing narrative text
- Rich contextual relationships
- Complex sentence structures
- Approximately 800 million words
English Wikipedia
The English Wikipedia extraction (excluding lists, tables, and headers) adds:
- Encyclopedic, factual content
- Broad topic coverage (science, history, culture, etc.)
- Approximately 2.5 billion words
Why These Corpora Matter
Think of BERT's pretraining like a child learning language by reading thousands of books and encyclopedias. The diversity is key:
- Volume: 3+ billion words provide enough examples to learn rare words and patterns
- Quality: Well-edited text (not social media noise) teaches proper grammar and structure
- Diversity: Books teach narrative flow; Wikipedia teaches factual descriptions
- Coverage: Together, they expose BERT to a wide vocabulary and countless contexts
This combination lets BERT learn general language understanding—not just domain-specific patterns. When you later fine-tune BERT on your specific task, it already "knows" how language works at a fundamental level.
Key Takeaway: BERT's power comes from pretraining on 3+ billion words from BookCorpus and Wikipedia—this massive, diverse corpus enables it to learn rich, general-purpose language representations that transfer effectively to downstream tasks.