Course contentsShow
← Machine Learning and Deep Learning
Lesson 1822 of 3,53838. Instruction Tuning and AlignmentFree lesson

Constitutional AI Phase 2: RL from AI Feedback

Training a preference model using AI-generated comparisons, then optimizing with RL.

Constitutional AI Phase 2: RL from AI Feedback

What you'll learn: How to train a preference model using AI-generated comparisons and optimize it with reinforcement learning—no human labelers required.

The Core Idea

In Constitutional AI Phase 1, you used an AI to critique and revise harmful outputs based on constitutional principles. Phase 2 takes this further: instead of using human preferences to train a reward model, you generate preference comparisons entirely with AI, then use those to train a preference model and run RL fine-tuning.

Think of it like this: You're teaching a student (your language model) by having a trusted teacher (your AI labeler) grade thousands of essay pairs. The student learns what "good" looks like without needing a human to read every single comparison.

How It Works

  1. Generate AI Preferences: Use your AI labeler (from Phase 1) to compare pairs of model responses. The labeler chooses which response better aligns with your constitutional principles.

  2. Train a Preference Model: Just like the reward model in standard RLHF, you train a preference model using the Bradley-Terry objective—but on AI-generated preference data instead of human labels.

  3. RL Fine-Tuning: Use the trained preference model as your reward signal in an RL algorithm (typically PPO or similar) to optimize your policy model, with a KL penalty to prevent drift.

Why This Matters

Constitutional AI Phase 2 combines the scalability of AI feedback with the alignment power of RL. You get the benefits of RLHF—nuanced optimization beyond supervised learning—without the bottleneck of human annotation. Your constitutional principles guide the entire process.

Key Takeaway: Constitutional AI Phase 2 completes the loop by training a preference model on AI-generated comparisons, then using RL to optimize the policy—achieving principled alignment at scale without human labelers.