Definition
Instead of baking safety implicitly into the weights, the model learns to recall and reason over the written safety spec in its chain of thought before answering (Guan et al., 2025).
- SFT on (prompt, CoT, output) examples whose CoT cites the spec; the spec is in the system prompt during generation, then stripped.
- RL with a reward from a judge LLM that sees the spec.
No human labels. o1 decodes a ROT13-hidden request, notices the trick and cites the policy; it improves jailbreak robustness and reduces overrefusal. The spec is written down in the Model Spec, with the chain of command Platform > Developer > User > Guideline.
Appears in
- Lecture 7, deliberative alignment
- Lecture 7, Pareto frontier
- Lecture 7, Model Spec
- Lecture 10, Anti-Scheming Training