Why RLHF is Needed: Limitations of Pretraining
What You'll Learn
You'll understand why even massive pretraining doesn't automatically make language models helpful, safe, or aligned with human values.
The Fundamental Gap
Pretraining teaches a model to predict the next token based on internet-scale text data. This objective—maximizing likelihood of the next word—creates powerful language understanding, but it has a critical blind spot: it doesn't distinguish between good and bad completions from a human perspective.
Think of it like learning to speak by reading everything ever written on the internet. You'd learn grammar, facts, and patterns—but you wouldn't automatically know what you should say in a conversation versus what you could say.
Three Key Limitations
1. Helpfulness Isn't Optimized
Pretraining learns "what comes next" in text, not "what would help the user most." A pretrained model might give a technically accurate but unhelpful rambling response when a concise answer was needed.
2. Harmlessness Isn't Captured
The internet contains toxic, biased, and harmful content. Pretraining on this data means the model learns these patterns too—there's nothing in the next-token objective that says "don't generate hate speech."
3. Preferences Are Missing
Different completions might all be plausible continuations, but humans prefer some over others. "I don't know" is better than a confident hallucination. Pretraining can't learn this—both are valid text patterns.
Why This Matters
A model that simply predicts internet text will sometimes be helpful, sometimes harmful, and often misaligned with what users actually want. This is where RLHF enters: it adds a training signal based on human preferences rather than just text likelihood.
Key Takeaway: Pretraining creates capability through next-token prediction, but cannot directly optimize for helpfulness, harmlessness, or human preferences—these require explicit alignment training like RLHF.