GPT-2: Scaling to 1.5B Parameters
You'll learn how dramatically increasing model size unlocked surprising zero-shot and few-shot capabilities that changed NLP forever.
The Giant Leap
GPT-1 had 117 million parameters. GPT-2 scaled up to 1.5 billion parameters—roughly 13 times larger. This wasn't just about making a bigger model; it was about discovering what happens when you cross certain scale thresholds.
What Changed at Scale
Think of it like the difference between a high school band and a full orchestra. The high schoolers can play well-rehearsed pieces (fine-tuned tasks), but the orchestra can sight-read new music on the fly (few-shot learning). GPT-2 revealed that sufficiently large language models develop an unexpected ability: they can perform tasks they weren't explicitly trained for, just from context.
Few-Shot Learning Emerges
GPT-2 demonstrated that by simply showing the model a few examples in the prompt (without any fine-tuning), it could:
- Translate languages
- Answer questions
- Summarize text
- Complete tasks it had never seen during training
This was revolutionary. Previous models like BERT needed task-specific fine-tuning. GPT-2 showed that scale + autoregressive pretraining = emergent generalization.
Why It Mattered
The researchers trained GPT-2 on a much larger, more diverse dataset (WebText) and discovered that simply continuing next-token prediction at massive scale created a model that understood tasks from prompts alone. The model wasn't given special training objectives—just more parameters, more data, and more compute.
Key Takeaway: GPT-2 proved that scaling model size from 117M to 1.5B parameters unlocked few-shot learning abilities, demonstrating that large autoregressive models can generalize to new tasks without fine-tuning—a paradigm shift in NLP.