Course contentsShow
← Machine Learning and Deep Learning
Lesson 1228 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Base Model Behavior: Completion vs Following Instructions

Why base models complete text rather than follow instructions, and how prompting differs from instruction-tuned models.

Base Model Behavior: Completion vs Following Instructions

You'll learn why base pretrained models complete your text rather than answer your questions, and how this differs from instruction-tuned models.

The Core Difference

When you use a base model like GPT-3 or GPT-2, you're working with a language model trained purely on next-token prediction. Its job is to complete patterns, not follow commands. If you type "Write a poem about dogs," a base model might continue with "...and cats and other animals that live in..." because it's completing what looks like a sentence fragment, not executing your instruction.

Think of it like autocomplete on your phone—it predicts what comes next based on patterns it's seen, but it doesn't understand you want it to do something for you.

Why This Happens

Base models learned by reading billions of web pages, books, and articles. They absorbed patterns like:

  • "The capital of France is Paris"
  • "Once upon a time there was..."
  • "Q: What is 2+2? A: 4"

They predict the next word based on statistical patterns, not semantic understanding of tasks.

Prompting vs Instructions

With base models, you must prompt carefully to trigger the right completion pattern. Instead of "Translate this to French," you might write:

"English: Hello. French: Bonjour. English: Goodbye. French:"

You're providing examples that guide completion, not giving direct instructions.

Instruction-tuned models (like ChatGPT) are fine-tuned specifically to interpret commands as tasks to execute, not patterns to complete. They've learned "when humans say 'write a poem,' they want me to generate original poetry, not continue their sentence."

Key Takeaway: Base models complete text based on statistical patterns from pretraining; they don't inherently follow instructions. Effective use requires prompting that leverages pattern completion, unlike instruction-tuned models that directly interpret commands.