Course contentsShow
← Machine Learning and Deep Learning
Lesson 3433 of 3,53875. LLM Safety and Alignment ChallengesFree lesson

Value Learning and Preference Uncertainty

Difficulty of specifying and learning complex human values, and handling uncertainty about what humans truly want.

Value Learning and Preference Uncertainty

What you'll learn: How AI systems struggle to capture what humans truly value and must handle deep uncertainty about our preferences.

The Core Challenge

Even if we solve reward hacking and specification gaming, a fundamental problem remains: we don't know how to precisely specify human values in the first place. Value learning is the ambitious goal of having AI systems learn what humans care about rather than hardcoding reward functions. But human preferences are complex, contradictory, context-dependent, and often we ourselves don't fully understand what we want.

Why This Is Hard

Consider teaching an AI to help you plan your career. Your "true preferences" involve tradeoffs between:

  • Income vs meaningful work
  • Stability vs growth opportunities
  • Work-life balance vs professional impact
  • Short-term satisfaction vs long-term fulfillment

You might give inconsistent answers depending on your mood, how questions are framed, or what you're currently experiencing. The AI faces preference uncertainty: it cannot be confident about what you actually value, only what you've expressed so far.

The Analogy

Imagine asking a chef to cook your "perfect meal" based on watching you eat a few times. They see you choose pizza over salad once, but was that your true preference or just convenience? Were you on a diet? Celebrating? The chef must remain uncertain and ideally ask clarifying questions rather than confidently optimizing for "maximum pizza."

Implications for Alignment

Advanced AI systems may need to:

  • Model uncertainty over multiple plausible human value systems
  • Ask for human feedback strategically when uncertainty is high
  • Avoid over-optimizing on noisy or incomplete preference signals
  • Handle value changes and learning over time

Key Takeaway: Value learning requires AI systems to infer complex human preferences from limited data while maintaining appropriate uncertainty, rather than confidently optimizing potentially misunderstood goals.