Course contentsShow
← Machine Learning and Deep Learning
Lesson 1202 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

GPT-2: Scaling to 1.5B Parameters

Examining the jump from 117M to 1.5B parameters and the emergence of few-shot learning capabilities.

GPT-2: Scaling to 1.5B Parameters

You'll learn how dramatically increasing model size unlocked surprising zero-shot and few-shot capabilities that changed NLP forever.

The Giant Leap

GPT-1 had 117 million parameters. GPT-2 scaled up to 1.5 billion parameters—roughly 13 times larger. This wasn't just about making a bigger model; it was about discovering what happens when you cross certain scale thresholds.

What Changed at Scale

Think of it like the difference between a high school band and a full orchestra. The high schoolers can play well-rehearsed pieces (fine-tuned tasks), but the orchestra can sight-read new music on the fly (few-shot learning). GPT-2 revealed that sufficiently large language models develop an unexpected ability: they can perform tasks they weren't explicitly trained for, just from context.

Few-Shot Learning Emerges

GPT-2 demonstrated that by simply showing the model a few examples in the prompt (without any fine-tuning), it could:

  • Translate languages
  • Answer questions
  • Summarize text
  • Complete tasks it had never seen during training

This was revolutionary. Previous models like BERT needed task-specific fine-tuning. GPT-2 showed that scale + autoregressive pretraining = emergent generalization.

Why It Mattered

The researchers trained GPT-2 on a much larger, more diverse dataset (WebText) and discovered that simply continuing next-token prediction at massive scale created a model that understood tasks from prompts alone. The model wasn't given special training objectives—just more parameters, more data, and more compute.

Key Takeaway: GPT-2 proved that scaling model size from 117M to 1.5B parameters unlocked few-shot learning abilities, demonstrating that large autoregressive models can generalize to new tasks without fine-tuning—a paradigm shift in NLP.