The Alignment Problem in LLMs
What you'll learn: Why large language models trained on human text can still produce harmful, biased, or misaligned outputs despite their training origins.
The Core Issue
Pre-trained LLMs learn from massive amounts of human-generated text scraped from the internet, books, and other sources. You might think this would naturally align them with human values and intentions—after all, they're learning from us. But here's the problem: human text data reflects both our best and worst qualities.
The internet contains helpful tutorials alongside hate speech, factual information alongside conspiracy theories, respectful discourse alongside toxic behavior. An LLM has no inherent mechanism to distinguish "good" from "bad" content—it simply learns statistical patterns from everything it sees.
Why This Happens
Think of an LLM like a mirror reflecting humanity's written expression. If that mirror captures graffiti alongside art, it doesn't know which is vandalism. The model learns:
- What is written, not what should be written
- Correlations and patterns, not moral principles
- Statistical likelihood, not ethical appropriateness
Even worse, certain biases and harmful patterns may be overrepresented in training data, making them statistically "normal" to the model.
The Misalignment
This creates misalignment: the model's behavior diverges from what we actually want. It might:
- Generate plausible-sounding misinformation
- Reproduce stereotypes and biases from training data
- Provide harmful instructions when prompted
- Optimize for engagement over truthfulness
Key Takeaway: Pre-trained LLMs inherit the full spectrum of human text—including harmful biases, toxicity, and misinformation—because they learn statistical patterns without inherent moral judgment. Alignment requires deliberate intervention beyond base training.