Course contentsShow
← AI Engineering
Lesson 1588 of 1,88638. Bias, Fairness, and AlignmentFree lesson

The Alignment Problem in LLMs

How pre-trained models can generate harmful, biased, or misaligned outputs despite training on human text data.

The Alignment Problem in LLMs

What you'll learn: Why large language models trained on human text can still produce harmful, biased, or misaligned outputs despite their training origins.

The Core Issue

Pre-trained LLMs learn from massive amounts of human-generated text scraped from the internet, books, and other sources. You might think this would naturally align them with human values and intentions—after all, they're learning from us. But here's the problem: human text data reflects both our best and worst qualities.

The internet contains helpful tutorials alongside hate speech, factual information alongside conspiracy theories, respectful discourse alongside toxic behavior. An LLM has no inherent mechanism to distinguish "good" from "bad" content—it simply learns statistical patterns from everything it sees.

Why This Happens

Think of an LLM like a mirror reflecting humanity's written expression. If that mirror captures graffiti alongside art, it doesn't know which is vandalism. The model learns:

  • What is written, not what should be written
  • Correlations and patterns, not moral principles
  • Statistical likelihood, not ethical appropriateness

Even worse, certain biases and harmful patterns may be overrepresented in training data, making them statistically "normal" to the model.

The Misalignment

This creates misalignment: the model's behavior diverges from what we actually want. It might:

  • Generate plausible-sounding misinformation
  • Reproduce stereotypes and biases from training data
  • Provide harmful instructions when prompted
  • Optimize for engagement over truthfulness

Key Takeaway: Pre-trained LLMs inherit the full spectrum of human text—including harmful biases, toxicity, and misinformation—because they learn statistical patterns without inherent moral judgment. Alignment requires deliberate intervention beyond base training.