Alignment Tradeoffs and Failure Modes
What you'll learn: How alignment constraints create tensions between helpfulness and safety, and where these safeguards can break down.
The Core Tension
Alignment isn't free—it comes with tradeoffs. When you make an LLM safer through techniques like RLHF, constitutional AI, or hard constraints, you often sacrifice some capability or user satisfaction. Think of it like childproofing a kitchen: locked cabinets prevent accidents but make cooking less convenient.
Helpfulness vs Safety
The primary tension exists between helpfulness (answering user requests fully) and safety (refusing harmful requests). Over-aligned models become "overly cautious," refusing legitimate requests because they pattern-match to risky scenarios. For example, a model might refuse to explain a historical war crime for educational purposes, or decline to debug security code even in appropriate contexts.
Over-Alignment Risks
Over-alignment (sometimes called "alignment tax") manifests as:
- Excessive refusals on benign requests
- Preachy or evasive responses that frustrate users
- Loss of nuance in sensitive but legitimate topics
- Reduced creativity or directness
This pushes users toward unaligned alternatives, defeating the safety purpose.
Common Failure Modes
Even well-aligned models fail predictably:
- Jailbreaks: Adversarial prompts bypass alignment constraints
- Context hijacking: Retrieval in RAG systems injects misaligned content
- Edge case brittleness: Unusual requests fall outside training distribution
- Misaligned objectives: The model optimizes for measured alignment metrics rather than true human values
Think of alignment like building safety rails on a mountain path—too low and people fall off, too high and they can't enjoy the view or climb over for legitimate reasons.
Key Takeaway: Alignment requires continuous balancing between safety and utility; no static solution perfectly handles all contexts, and over-correction creates its own risks that undermine trust and adoption.