Definition
A jailbreak is an input that bypasses an LLM’s safety training so that it produces content it should refuse. The user is the attacker (deliberate misuse), unlike in a prompt injection.
| Method | Type | Key idea |
|---|---|---|
| GCG | white-box | gradient-optimized adversarial suffix |
| PAIR | black-box | an attacker LLM refines jailbreaks |
| Crescendo | multi-turn | gradual escalation across turns |
| Many-shot | in-context | 100s of harmful examples in the prompt |
No LLM is immune: a sufficiently motivated attacker can bypass safety training, though it is becoming harder over time. Jailbreaks are the LLM version of adversarial examples in vision.
Appears in
- Lecture 1, Deliberate Misuse
- Lecture 2, Sampling Matters for Safety: decoding parameters as a jailbreak
- Lecture 3, Jailbreaking: definition, two failure modes, all attack methods
- Lecture 3, Two Main Mechanisms of Jailbreaks
- Lecture 4, A Few Harmful Examples Erase Refusal: fine-tuning as a jailbreak for open weights
Related concepts
- Prompt Injection
- Emergent Misalignment (fine-tuning as a jailbreak)