GPT-2's Zero-Shot Task Transfer
What you'll learn: How GPT-2 could perform tasks it was never explicitly trained for, simply by understanding natural language instructions.
The Revolutionary Discovery
GPT-2 surprised the AI community by demonstrating that a language model could perform various NLP tasks without any fine-tuning — just by reading task instructions in plain English. This is called zero-shot learning: the model performs a task with zero training examples for that specific task.
How It Works
Remember that GPT-2 was trained purely on next-token prediction across massive amounts of text. During this pretraining, it accidentally learned to follow patterns like:
- "Translate to French: Hello → Bonjour"
- "Summarize: [long text] TL;DR: [summary]"
- "Q: What is the capital? A:"
When you give GPT-2 a prompt (task description in natural language), it simply continues the text pattern it learned during pretraining. You're not training it on a new task — you're conditioning its generation on the format of that task.
The Analogy
Think of a student who's read thousands of textbooks. Even without taking a specific test prep course, they can attempt new question formats because they've seen similar patterns before. GPT-2 works similarly: it recognizes task structures from its broad reading.
Why This Matters
Previous models like BERT required fine-tuning (retraining the model's weights) for each new task. GPT-2 showed that with enough scale and data, autoregressive models could adapt through prompting alone — no parameter updates needed. This opened the door to viewing language models as general-purpose text processors.
Key Takeaway: GPT-2's zero-shot capability demonstrated that autoregressive pretraining at scale enables task performance through natural language prompts alone, without fine-tuning the model's parameters.