Base Models: Pretraining Objective and Capabilities
What you'll learn: You'll understand what base models learn during pretraining and what they can (and cannot) do out of the box.
What Is a Base Model?
A base model is a language model fresh off pretraining—before any fine-tuning, instruction tuning, or RLHF. Think of it like a brilliant student who has read every book in the library but hasn't yet learned how to follow directions or hold a conversation.
The Pretraining Objective
Base models are trained on a simple but powerful objective: predict the next token (for decoder-only models like GPT) or reconstruct masked/corrupted text (for encoder-decoder models like T5 and BART).
This raw next-token prediction teaches the model:
- Grammar and syntax
- Factual knowledge from training data
- Patterns in language structure
- Statistical relationships between words and phrases
Raw Capabilities
Out of the box, base models can:
- Complete text that you start ("The capital of France is...")
- Continue patterns they've seen during training
- Generate coherent text in the style of their training data
However, they cannot:
- Follow instructions reliably ("Write a poem about...")
- Engage in helpful dialogue
- Refuse inappropriate requests
- Format answers consistently
Analogy: A base model is like an autocomplete engine on steroids—it predicts what comes next based on massive pattern recognition, but it doesn't "understand" what you want it to do.
Why This Matters
Understanding base models helps you appreciate why modern systems like ChatGPT require additional training (instruction tuning, RLHF) beyond pretraining. The base model provides the linguistic foundation; subsequent training teaches it to be helpful, harmless, and honest.
Key Takeaway: Base models excel at next-token prediction and pattern completion but lack instruction-following capabilities until further trained—they're linguistic engines, not assistants.