Course contentsShow
← Machine Learning and Deep Learning
Lesson 1763 of 3,53838. Instruction Tuning and AlignmentFree lesson

Why RLHF is Needed: Limitations of Pretraining

How pretraining objectives don't capture helpfulness, harmlessness, or human preferences directly.

Why RLHF is Needed: Limitations of Pretraining

What You'll Learn

You'll understand why even massive pretraining doesn't automatically make language models helpful, safe, or aligned with human values.

The Fundamental Gap

Pretraining teaches a model to predict the next token based on internet-scale text data. This objective—maximizing likelihood of the next word—creates powerful language understanding, but it has a critical blind spot: it doesn't distinguish between good and bad completions from a human perspective.

Think of it like learning to speak by reading everything ever written on the internet. You'd learn grammar, facts, and patterns—but you wouldn't automatically know what you should say in a conversation versus what you could say.

Three Key Limitations

1. Helpfulness Isn't Optimized

Pretraining learns "what comes next" in text, not "what would help the user most." A pretrained model might give a technically accurate but unhelpful rambling response when a concise answer was needed.

2. Harmlessness Isn't Captured

The internet contains toxic, biased, and harmful content. Pretraining on this data means the model learns these patterns too—there's nothing in the next-token objective that says "don't generate hate speech."

3. Preferences Are Missing

Different completions might all be plausible continuations, but humans prefer some over others. "I don't know" is better than a confident hallucination. Pretraining can't learn this—both are valid text patterns.

Why This Matters

A model that simply predicts internet text will sometimes be helpful, sometimes harmful, and often misaligned with what users actually want. This is where RLHF enters: it adds a training signal based on human preferences rather than just text likelihood.

Key Takeaway: Pretraining creates capability through next-token prediction, but cannot directly optimize for helpfulness, harmlessness, or human preferences—these require explicit alignment training like RLHF.