Adversarial Example

Definition

An adversarial example $x_{\text{adv}}=x+{\delta }^{\star }$ with ${\delta }^{\star }=\arg {\max }_{\Vert \delta {\Vert }_p\le \epsilon }\mathcal{L}(f(x+\delta ),y)$: the worst input inside a small ball around a clean input. The threat model fixes the norm (${\ell }_{\infty }$, ${\ell }_2$, ${\ell }_0$) and the budget $\epsilon$; for LLMs it becomes e.g. “a suffix of $k$ tokens”.

• Training minimizes the average loss, the attacker picks the worst case: 95% clean accuracy can sit next to ~0% robust accuracy.
• FGSM: one step $x+\epsilon \text{sign}({\nabla }_x\mathcal{L})$. PGD: many projected sign-steps, near worst case.
• Perturbations are often universal (one $\delta$ for many inputs) and transferable (one $\delta$ for many models), and work physically (stop-sign stickers).

Appears in

• Lecture 3, average vs. worst case
• Lecture 3, threat models
• Lecture 3, FGSM
• Lecture 3, PGD
• Lecture 3, universal and transferable

Related concepts

• Adversarial Training
• Adaptive Attack
• Jailbreak