Course contentsShow
← Machine Learning and Deep Learning
Lesson 1821 of 3,53838. Instruction Tuning and AlignmentFree lesson

Constitutional AI Phase 1: Critique and Revision

Using the model to critique its own outputs against principles and generate improved responses.

Constitutional AI Phase 1: Critique and Revision

What You'll Learn

You'll learn how to have a language model improve its own responses by critiquing them against explicitly defined principles before any human feedback is collected.

The Core Idea

Constitutional AI Phase 1 is a self-improvement loop where the model acts as both student and teacher. Instead of immediately collecting human preferences, you first let the model refine its own outputs using a "constitution"—a set of guiding principles written in natural language.

Here's how it works:

  1. Generate an initial response to a prompt (often a harmful or problematic one)
  2. Critique that response using constitutional principles (e.g., "Is this response helpful and harmless?")
  3. Revise the response based on the critique to better align with the principles
  4. Repeat this critique-revision cycle if needed

Think of it like having a friend review your essay draft and suggest improvements—except the model is both the writer and the reviewer, guided by ethical rules you've written down.

Why This Matters

This phase creates training data automatically without requiring thousands of human annotators. The model generates pairs of (original, revised) responses where the revised versions better follow your constitutional principles. These improved responses become training examples for supervised fine-tuning.

The constitution itself might include principles like "Choose the response that is less harmful" or "Prefer responses that respect human autonomy." The model uses these as evaluation criteria during its self-critique.

Connection to Earlier Concepts

This builds directly on the Constitutional AI principles framework you just learned (lesson 1820). Phase 1 generates the improved dataset; later phases use this data for training (DPO or RLHF-style methods).

Key Takeaway: Constitutional AI Phase 1 uses the model's own reasoning to critique and revise outputs against explicit principles, creating aligned training data without initial human labeling.