Value Learning and Preference Uncertainty
What you'll learn: How AI systems struggle to capture what humans truly value and must handle deep uncertainty about our preferences.
The Core Challenge
Even if we solve reward hacking and specification gaming, a fundamental problem remains: we don't know how to precisely specify human values in the first place. Value learning is the ambitious goal of having AI systems learn what humans care about rather than hardcoding reward functions. But human preferences are complex, contradictory, context-dependent, and often we ourselves don't fully understand what we want.
Why This Is Hard
Consider teaching an AI to help you plan your career. Your "true preferences" involve tradeoffs between:
- Income vs meaningful work
- Stability vs growth opportunities
- Work-life balance vs professional impact
- Short-term satisfaction vs long-term fulfillment
You might give inconsistent answers depending on your mood, how questions are framed, or what you're currently experiencing. The AI faces preference uncertainty: it cannot be confident about what you actually value, only what you've expressed so far.
The Analogy
Imagine asking a chef to cook your "perfect meal" based on watching you eat a few times. They see you choose pizza over salad once, but was that your true preference or just convenience? Were you on a diet? Celebrating? The chef must remain uncertain and ideally ask clarifying questions rather than confidently optimizing for "maximum pizza."
Implications for Alignment
Advanced AI systems may need to:
- Model uncertainty over multiple plausible human value systems
- Ask for human feedback strategically when uncertainty is high
- Avoid over-optimizing on noisy or incomplete preference signals
- Handle value changes and learning over time
Key Takeaway: Value learning requires AI systems to infer complex human preferences from limited data while maintaining appropriate uncertainty, rather than confidently optimizing potentially misunderstood goals.