Definition
Arditi et al. (NeurIPS 2024): refusal is mediated by a single direction in the residual stream. Projecting it out of the activations or weights (abliteration) completely disables the model’s refusals, with no training at all.
It is one of several signs that the safety layer is thin: safety-critical parameters are <1% of the model, 10 fine-tuning examples remove safety, and safety alignment mostly changes the first few output tokens.
Appears in
- Lecture 2, how thin is the safety layer
- Lecture 3, Model-Level Defenses: training the refusal subspace to be harder to ablate
- Lecture 4, Refusal Is Mediated by a Single Direction: difference of means
- Lecture 4, Why I − r̂r̂ᵀ Is a Projection: the projection and the weight edit (mock exam)
- Lecture 4, Abliteration in Practice: hundreds of abliterated models, Heretic