Definition
AI control assumes the model itself may be scheming or misaligned and designs protocols that stay safe even if it actively tries to subvert them. It complements alignment instead of relying on it.
- Setup: a powerful but untrusted model U does the work, a weaker but trusted model T monitors, edits or takes over, and a small human auditing budget H (~2%) is available.
- Protocols: trusted monitoring, untrusted monitoring, trusted editing, defer-to-trusted, upfront auditing.
- Evaluation: a red team plays the untrusted model. Safety (no backdoor is ever submitted) is scored against usefulness (task performance kept).
| Protocol (Greenblatt et al., ICML 2024) | Useful | Safe |
|---|---|---|
| trusted only | 2% | 100% |
| upfront auditing | 100% | 15% |
| trusted monitoring | 97% | 62% |
| untrusted monitoring | 96% | 79% |
| defer to trusted | 93% | 83% |
| trusted editing | 94% | 92% |
Realistic settings show that capable agents still slip through:
- BashArena (root sysadmin tasks): 26% undetected sabotage.
- LinuxArena (live production systems): ~34% undetected sabotage.
- ResearchArena: sabotage embedded in post-training artifacts is near-undetectable (AUC 0.51 to 0.63).
Appears in
- Lecture 1, AI as the Adversary
- Lecture 10, Why This Topic Is Genuinely Hard: why we need control
- Lecture 12, the control setup
- Lecture 12, protocols
- Lecture 12, safety-usefulness frontier
- Lecture 12, BashArena, LinuxArena, ResearchArena