Definition
Constitutional AI (Bai et al., 2022) trains a helpful, harmless and honest model with less human feedback. A written set of principles, the constitution, guides self-improvement.
- Supervised phase: the model critiques its own responses against the principles and revises them; the revisions become training data.
- RLAIF phase: an AI model judges which responses are better; Reinforcement Learning from AI Feedback replaces human labeling.
Result: less harmful and less evasive than standard RLHF, and the values are transparent and auditable.
Appears in
- Lecture 1, RLHF and Constitutional AI
- Lecture 3, Model-Level Defenses
- Lecture 6, Three Interventions: bio-targeted constitutional classifiers
- Lecture 7, The Two-Phase CAI Pipeline: SL-CAI and RLAIF in detail