Definition
An adaptive attack is built for exactly the defense it targets. Because the defender publishes first and the attacker moves second, static benchmark numbers (measured against fixed, older attacks) are only an upper bound on robustness.
- Obfuscated gradients: 7 of 9 ICLR 2018 defenses fell to near 0% (Athalye et al., 2018).
- Nasr et al. (2025): 12 of 12 recent LLM defenses broken, from near 0% to >90% attack success, with ordinary tools (gradients, RL, random search, humans).
- Kerckhoffs’s principle: assume the attacker knows the system; don’t rely on secrecy.
Rule
A defense is only as strong as the strongest adaptive attack tried against it.
Appears in
- Lecture 3, gradient masking
- Lecture 3, Kerckhoffs
- Lecture 3, the attacker moves second
- Lecture 4, …But TAR Is Not Durable Either: adaptive fine-tuning breaks tamper resistance
- Lecture 9, LLMail-Inject: 208K crowd attacks