GCG

Formula

$$\underset{s\in {\mathcal{V}}^k}{\min }-\log P_{\theta }(t^{\star }\mid x\oplus s)$$

$s$ = suffix of $k$ tokens, $t^{\star }$ = target prefix “Sure, here is…“.

Per iteration: take the gradient w.r.t. the one-hot token embeddings, keep the top-K candidate tokens per position, evaluate a batch of one-token swaps with forward passes, keep the best. It is discrete PGD: gradients propose, forward passes decide.

Why it works: safety training mostly shapes the first answer tokens; once “Sure, here is” is likely, autoregression continues. Suffixes found on open models transfer to ChatGPT, Claude, Bard. AmpleGCG trains a generator on GCG outputs.

Appears in

• Lecture 3, GCG
• Lecture 3, specialized attacker models

Related concepts

• Jailbreak
• PAIR
• Adversarial Example