Course contentsShow
← Machine Learning and Deep Learning
Lesson 1206 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

In-Context Learning: Learning from Examples

How GPT-3 performs tasks by conditioning on examples in the prompt without parameter updates.

In-Context Learning: Learning from Examples

You'll learn: How GPT-3 performs tasks by conditioning on examples in the prompt without updating its parameters.

What Is In-Context Learning?

In-context learning is GPT-3's remarkable ability to learn new tasks on the fly just by seeing a few examples in the prompt—no fine-tuning or gradient updates required. You simply show the model what you want in the input text itself, and it adapts its behavior accordingly.

Think of it like showing a chef a few plated dishes at a restaurant before asking them to prepare your order. They don't need weeks of training—they can see the pattern and replicate the style immediately.

How It Works

With GPT-3's massive scale (175B parameters), the model has internalized so many patterns during pretraining that it can recognize task structures from just a few demonstrations. You provide:

  1. Task examples in the prompt (e.g., "English: Hello → French: Bonjour")
  2. A new input following the same format
  3. The model completes the pattern using its autoregressive prediction

This is called "few-shot learning" when you provide several examples, "one-shot" with a single example, or even "zero-shot" when you just describe the task in natural language.

Why This Matters

Traditional machine learning requires collecting labeled data, training/fine-tuning models, and managing separate model versions for each task. In-context learning lets you adapt GPT-3 to new tasks instantly by crafting clever prompts—no parameter updates needed. The knowledge is already encoded; you're just steering it with examples.

The limitation? It only works within the context window and relies on prompt quality. But it fundamentally changed how we interact with language models.

Key Takeaway: In-context learning allows GPT-3 to perform new tasks by conditioning on examples provided directly in the prompt, leveraging patterns learned during pretraining without any parameter updates.