Course contentsShow
← Machine Learning and Deep Learning
Lesson 1149 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

BERT Pretraining Data: BookCorpus and Wikipedia

The large-scale corpora used to pretrain BERT and their importance for general language understanding.

BERT Pretraining Data: BookCorpus and Wikipedia

What You'll Learn

You'll understand the massive text corpora used to pretrain BERT and why their size and diversity are crucial for learning rich language representations.

The Foundation: Where BERT Learns Language

BERT wasn't trained on random internet text—its creators carefully selected two large-scale datasets that together provided over 3 billion words of high-quality English text:

BookCorpus

The BookCorpus dataset contains over 11,000 unpublished books spanning diverse genres: romance, fantasy, adventure, science fiction, and more. This corpus provides:

  • Natural, flowing narrative text
  • Rich contextual relationships
  • Complex sentence structures
  • Approximately 800 million words

English Wikipedia

The English Wikipedia extraction (excluding lists, tables, and headers) adds:

  • Encyclopedic, factual content
  • Broad topic coverage (science, history, culture, etc.)
  • Approximately 2.5 billion words

Why These Corpora Matter

Think of BERT's pretraining like a child learning language by reading thousands of books and encyclopedias. The diversity is key:

  1. Volume: 3+ billion words provide enough examples to learn rare words and patterns
  2. Quality: Well-edited text (not social media noise) teaches proper grammar and structure
  3. Diversity: Books teach narrative flow; Wikipedia teaches factual descriptions
  4. Coverage: Together, they expose BERT to a wide vocabulary and countless contexts

This combination lets BERT learn general language understanding—not just domain-specific patterns. When you later fine-tune BERT on your specific task, it already "knows" how language works at a fundamental level.

Key Takeaway: BERT's power comes from pretraining on 3+ billion words from BookCorpus and Wikipedia—this massive, diverse corpus enables it to learn rich, general-purpose language representations that transfer effectively to downstream tasks.