Self-Critique and Revision
What you'll learn: How to train language models to evaluate and improve their own outputs by comparing them against constitutional principles.
The Core Idea
Self-critique and revision is a powerful extension of Constitutional AI where the model acts as its own reviewer. Instead of simply generating a response and stopping, the model learns to:
- Generate an initial response
- Critique that response against constitutional principles (honesty, harmlessness, helpfulness, fairness)
- Identify specific problems or violations
- Revise the response to address those issues
Think of it like writing an essay with built-in self-editing: you draft your first version, review it against your standards, spot the weaknesses, and rewrite before submitting.
How It Works in Practice
After Constitutional AI training (lesson 1590), models can be prompted or fine-tuned to perform explicit critique steps. For example:
- Step 1 (Generate): The model produces an initial answer
- Step 2 (Critique): The model examines its own output: "Does this response contain stereotypes? Is it unnecessarily evasive? Does it make unsupported claims?"
- Step 3 (Revise): Based on identified issues, the model generates an improved version
This creates a feedback loop within a single generation process. The model essentially debates with itself, catching bias, errors, or alignment failures before they reach the user.
Benefits
- Reduced harmful outputs without external filtering
- Improved reasoning transparency (you can log the critique)
- Scalable alignment that doesn't require constant human review
- Adaptive correction based on constitutional principles rather than rigid rules
Key Takeaway: Self-critique and revision transforms models into self-monitoring systems that can identify and fix alignment issues in their own outputs before delivering final responses.