Constitutional AI: Principles-Based Alignment
You'll learn how to encode human values as explicit principles that guide AI behavior through self-critique and revision.
Core Idea
Constitutional AI (CAI) takes the AI labeler concept from RLAIF one step further: instead of just having an AI provide preference labels, you give it a constitution—a written set of principles or values—and train it to critique and revise its own outputs according to those principles.
Think of it like teaching a student to self-edit their essay using a rubric. Rather than waiting for a teacher's feedback on every draft, the student learns the grading criteria and applies them independently.
How It Works
The process has two main phases:
1. Supervised Learning Phase (Self-Critique)
- The model generates an initial response
- It then critiques that response against constitutional principles (e.g., "Is this response harmful?", "Does it respect privacy?")
- It revises its own output based on the critique
- The revised responses are used for supervised fine-tuning
2. RL Phase (Preference Modeling)
- The AI generates multiple responses to prompts
- Using the constitutional principles, the AI itself judges which responses better follow the constitution
- These AI-generated preferences replace human labels in the RLHF pipeline
- DPO or PPO training proceeds using these principle-based preferences
The constitution might include principles like "Choose the response that is most helpful and harmless" or "Avoid responses that could enable illegal activities."
Key Takeaway: Constitutional AI replaces implicit human values with explicit, written principles that the model uses to critique itself and generate training preferences, making alignment more transparent, scalable, and controllable than pure human feedback.