Lecture 4: Open-Weight LLM Safety

TL;DR

1. The gap is small: the open-weight frontier trails the closed one by 3 to 6 months, and distillation attacks (training on millions of API answers) close it faster.
2. Fine-tuning breaks safety: 10 harmful examples ($0.20) remove refusal from GPT-3.5 Turbo; identity-shift data and even benign data degrade it, because fine-tuning shifts the output prefix towards “Sure, here is…“.
3. Abliteration: refusal is one direction $\hat{r}=({\mu }_{\text{harmful}}-{\mu }_{\text{harmless}})/\Vert \cdot \Vert$ in the residual stream. Projecting it out with $(I-\hat{r}{\hat{r}}^{\top })$, also baked into the weights, removes refusal without training. It is representation engineering with adversarial intent, and now a one-command download.
4. Emergent misalignment: narrow harmful fine-tuning (insecure code, bad medical or financial advice) produces broadly misaligned models; framing the same code as educational prevents it. Reward hacking in production RL causes the same, including alignment faking and sabotage.
5. Defenses for open weights (RMU unlearning on WMDP, TAR tamper-resistance) work against the attacks they were built for but fall to adaptive fine-tuning and direction removal. Open-weight policy has no clean answer; it depends on the risk class.

Exam relevance

3 questions from this lecture. Mock exam question: Abliteration: state (a) what is removed, (b) the linear-algebra operation, (c) whether training is required (answer in Why I − r̂r̂ᵀ Is a Projection). Likely topics: why fine-tuning on few or benign examples breaks safety, the refusal direction and the projection math, emergent misalignment and the educational-insecure control, EM from reward hacking, LoRA, why RMU and TAR fail under adaptive attacks, the policy trade-off of open weights. Emergent misalignment is also the topic of Mini-Project 2.

Overview: I. Wrap-up of Lecture 3, II. Open vs. closed weights, III. Fine-tuning breaks safety, IV. Abliteration, V. Emergent misalignment, VI. Defenses for open weights.

Today

Slide 2

Part	Topics
I. Wrap-up of Lecture 3	Crescendo, decomposition, automated attack discovery; Swiss-cheese, model- and system-level defenses
II. Open vs. closed gap	how far open-weight models trail proprietary ones; why distillation closes the gap
III. Fine-tuning breaks safety	even benign data degrades alignment; a few harmful samples remove refusal
IV. Abliteration	refusal lives in a single direction; targeted, training-free removal
V. Emergent misalignment	narrow harmful fine-tuning produces broadly misaligned models; reward hacking does the same
VI. Tamper-resistance and unlearning	WMDP, RMU, TAR; none robust under adaptive attack

Mini-Project 2, Part 1 is based on Part V only: reproduce emergent misalignment by fine-tuning a small open-weight LLM with LoRA.

I. Wrap-up of Lecture 3

Recap: Attacks

Slide 3

Attack	Access	One line
Prefilling	white-box / API	force the answer to start with “Sure, here is…”; 100% on every model that exposes the prefix
Best-of-N	black-box	random augmentations; 10K samples reach 89% on GPT-4o; power law in $N$
Many-shot	long context	256 fake compliant Q&A pairs; in-context compliance
GCG	white-box	greedy coordinate gradient over a $k$-token suffix; discrete PGD on the refusal-prefix logprob
Random search	score-based	hill-climbing on the logprob with one-token swaps; 100% on Llama-2/3, GPT-3.5/4o, Gemma, Phi-3
PAIR	black-box	attacker LLM rewrites the prompt with judge feedback; ~20 queries, ~$0.03

Full explanations: Lecture 3, Attack Methods.

Crescendo in Practice

Slide 4

The direct ask is refused. Then start with a benign reference and ladder up: T1 history and context, T2 chemical principles, T3 combine into a how-to, T4 more detail. Crescendomation beats PAIR, many-shot and CIA by 29% to 61% on GPT-4, usually in ≤ 5 turns (Russinovich, Salem & Eldan, USENIX Security 2025).

ChatGPT

Gemini Ultra

A direct ask is refused (top of each panel); five benign-looking follow-ups produce the harmful answer. See Lecture 3, Crescendo.

Decomposition and Specialized Attacker Models

Slides 5-6

Decomposition (slide 5) splits a malicious task into innocuous sub-tasks; a state-backed group automated 80% to 90% of an espionage campaign with Claude Code this way (Anthropic, “Disrupting AI espionage”; details and figure in Lecture 3).

Specialized attacker models (slide 6):

• AmpleGCG (Liao et al., 2024; AmpleGCG-Plus): a generator trained on GCG outputs, hundreds of suffixes per prompt in minutes.
• RL is a hammer (Wen et al., 2025): the attacker is an RL policy with the judge score as reward. With a diversity reward (BLEU- or embedding-based), one policy generates many distinct, fluent injections instead of repeating one template.

Cost of a trained attacker

Once the specialized model is trained, each new attack costs a single forward pass, unlike GCG, which typically needs thousands of queries.

Slide 6: two RL-generated injections per reward, all aimed at "unlock my front door". BLEU and embedding rewards push the policy towards different phrasings. Source: Wen et al., 2025.

Automated Attack Discovery Using LLM Agents

Slide 7

Claude Code (Opus 4.6) in an autonomous loop proposes, implements, runs and evaluates attack variants; after 56 iterations its best attack reaches 100% on Meta SecAlign-70B vs. 56% for the best human baseline (Panfilov et al., 2026). Figure and details: Lecture 3.

Defenses Recap

Slides 8-12

• Swiss cheese (slide 9): stack independent, imperfect layers so their holes don’t line up (International AI Safety Report 2026, Fig. 3.5); see Lecture 3. Operational view: a defense only needs to be expensive enough that a casual attacker (20 minutes, $0.10) gives up; for catastrophic risk (CBRN, mass cyber-offense) it must hold against state actors, and we are not there yet.
• Model-level (slide 10): RLHF (Ouyang et al., 2022), Constitutional AI (Bai et al., 2022), deliberative alignment (Guan et al., 2024), instruction hierarchy (Wallace et al., 2024), adversarial training with hard-negative mining (Mazeika et al., CAIS 2024) and, new on this slide, representation engineering: modify the model’s internal representations to disrupt harmful outputs (Zou et al., 2023).
• System-level (slide 11): permission gating, input/output classifiers (Constitutional Classifiers: 86% → 4.4%), activation probes, CaMeL.
• More capable, more robust? (slide 12): for indirect prompt injection yes, Mythos Preview ~0.1% even at $k=100$ (system card, Gray Swan ART); but capable models know more dangerous content.

Take-Home of Lectures 3 and 4 so far

Slide 13

1. The attacker moves second: you need adaptive attacks designed for a specific defense.
2. Pixels vs. tokens: many lessons from vision transfer to LLMs.
3. Two failure modes: competing objectives and mismatched generalization.
4. Universal and transferable: one suffix can transfer to proprietary models.
5. Swiss cheese: stack defenses, none is perfect.
6. Capable and consequential: more robust to prompt injection, but deployed in higher-stakes settings.

Today's shift

Everything above assumed a deployed API. With open weights, the attacker has the model itself: fine-tuning, weight editing and activation surgery become possible. That is a new attack surface.

II. Open vs. Closed Weights

Slide 14

The Capability Gap, Two Views

Slide 15

The open-weight frontier (DeepSeek V3.1, Qwen3, Kimi K2, GPT-OSS, Llama-3.x) trails the closed frontier by 3 to 6 months. Once a closed model ships a capability, an open imitation typically appears within one release cycle.

US (closed) vs. PRC (mostly open) frontier trajectories, 2024 to 2026. Source: U.S. CAISI.

Artificial Analysis Intelligence Index: open weights within ~5 points of the closed frontier by early 2026. Source: Artificial Analysis v4.0.

One Way to Close It: Distillation Attacks

Slide 16

Distillation attack

Open many API accounts, query Claude or GPT at scale, and train a student model on the (prompt, response) pairs. The student inherits much of the teacher’s capability for a fraction of the pre-training cost.

• Anthropic attributed 16M+ exchanges across 24K fraudulent accounts to DeepSeek, Moonshot and MiniMax (Nov 2025).
• Detection signals: behavioral fingerprinting, rare-pair triggers, output style, clustering of the account graph (Anthropic, “Detecting and preventing distillation attacks”).

Slide 16: Anthropic's overview of distillation attacks. Source: Anthropic, 2025.

III. Fine-Tuning Breaks Safety

A Few Harmful Examples Erase Refusal

Slide 17

Qi, Zeng, Xie, Chen, Jia, Mittal & Henderson, “Fine-tuning Aligned Language Models Compromises Safety…”, ICLR 2024 fine-tune GPT-3.5 Turbo and Llama-2 and grade harmfulness across 11 OpenAI/Meta usage-policy categories:

• 10 harmful examples reach near-maximum harmfulness on GPT-3.5 Turbo via the OpenAI fine-tuning API, for about $0.20.
• Identity-shift data (no harmful content at all) also breaks safety: train the model to always say “Sure”, and it generalizes.
• Even Alpaca (benign instruction tuning) raises harmfulness measurably.

The three datasets: harmful, identity shift, benign.

Harmfulness (1 to 5) in 11 categories before (gray) and after (red) fine-tuning. Source: Qi et al., 2024.

What in “Safe” Data Actually Breaks Safety?

Slide 18

He, Xia & Henderson, “What is in Your Safe Data?”, COLM 2024 take the Alpaca finding apart. Even in a fully benign corpus, some examples are far more dangerous than others:

• Score each candidate by (i) representation and (ii) gradient proximity to known harmful examples, and distance from known safe ones.
• The top-scoring 100 benign samples raise GPT-3.5’s harmfulness 3× compared to a random 100.
• The dangerous benign examples are list-style or instruction-following Q&A: they teach the model to say “Sure, here is…“.

Mechanism

Fine-tuning shifts the prefix distribution towards affirmative compliance, whatever the content. Refusal lives in a thin layer (Lecture 2); any tuning that thins it further leaks.

Implication for fine-tuning APIs

For closed-API fine-tuning services (OpenAI, Google, Anthropic), “safe data” filters are not enough: the selected examples can erode safety even if every single line is innocuous.

IV. Abliteration

Slide 19

Refusal Is Mediated by a Single Direction

Slide 20

Arditi, Obeso, Syed, Paleka, Panickssery, Gurnee & Nanda, NeurIPS 2024 study 13 open-weight chat models up to 72B. For each they extract one direction $\hat{r}\in {\mathbb{R}}^{d_{\text{model}}}$ in the residual-stream activation space:

Refusal direction (difference of means)

$$\hat{r}=\frac{{\mu }_{\text{harmful}}-{\mu }_{\text{harmless}}}{\Vert {\mu }_{\text{harmful}}-{\mu }_{\text{harmless}}\Vert }$$

${\mu }_{\text{harmful}}$, ${\mu }_{\text{harmless}}$ = mean activations at one layer $\ell$ over harmful and harmless prompts.

Two interventions:

• Addition: add $\hat{r}$ to the activations of a harmless prompt ⇒ the model refuses harmless requests.
• Directional ablation: remove $\hat{r}$ from the activations of harmful prompts ⇒ the model complies with harmful requests.

Slide 20: activation addition (eq. 3) and directional ablation (eq. 4). Source: Arditi et al., 2024.

Why I − r̂r̂ᵀ Is a Projection

Slide 21

Directional ablation ( $\Vert \hat{r}\Vert =1$)

$$x^{\prime }=x-\hat{r}{\hat{r}}^{\top }x=(I-\hat{r}{\hat{r}}^{\top })x$$

${\hat{r}}^{\top }x$ is the length of the component of $x$ along $\hat{r}$; subtracting $\hat{r}({\hat{r}}^{\top }x)$ removes exactly that component.

Projection check ($P^2=P$):

$$\begin{array}{rl}(I-\hat{r}{\hat{r}}^{\top }{)}^2& =I-2\hat{r}{\hat{r}}^{\top }+\hat{r}({\hat{r}}^{\top }\hat{r}){\hat{r}}^{\top }\\ & =I-2\hat{r}{\hat{r}}^{\top }+\hat{r}{\hat{r}}^{\top }=I-\hat{r}{\hat{r}}^{\top }\end{array}$$

using ${\hat{r}}^{\top }\hat{r}=1$. It projects onto the hyperplane ${\hat{r}}^{\perp }=\{v:{\hat{r}}^{\top }v=0\}$: the component along $\hat{r}$ is zeroed and stays zero (applying it again changes nothing).

Slide 21: black = activation x, red = refusal direction r̂, blue = x′ = (I − r̂r̂ᵀ)x on the hyperplane r̂⊥.

Equivalent weight edit

$$W\leftarrow (I-\hat{r}{\hat{r}}^{\top })W$$

for every output projection that writes into the residual stream (embedding, attention output, MLP output). Then no component along $\hat{r}$ can ever be written into the stream. No backprop, no data except what is needed to compute $\hat{r}$, no fine-tuning.

Mock exam: abliteration

(a) What is removed: the single refusal direction $\hat{r}$ in the residual stream. (b) The operation: directional ablation, i.e. the orthogonal projection $(I-\hat{r}{\hat{r}}^{\top })$ applied to the activations, or baked into the weights as $W\leftarrow (I-\hat{r}{\hat{r}}^{\top })W$. (c) No training is required: it is a training-free weight edit. See Refusal Direction.

Bigger Picture: Representation Engineering (RepE)

Slide 22

Zou et al., “Representation Engineering: A Top-Down Approach to AI Transparency”, 2023:

• Reading: linear probes recover directions for hallucination, honesty, power, morality from activations.
• Control: adding or ablating those directions steers behavior.

Reading: probes find concept directions in the activations.

Control: adding or ablating the directions steers behavior. Source: Zou et al., 2023.

Intuition

Refusal is one instance of a general pattern. Honesty, deception, sycophancy, hallucination, power-seeking, even fact recall all admit (approximately) linear probes and interventions in the residual stream (Linear Probe). Abliteration is RepE applied with adversarial intent. Steering vectors come back in Lecture 7.

Abliteration in Practice: HuggingFace

Slide 23

• Training-free: no backprop, no GPU minutes. Compute $\hat{r}$ from a few hundred prompts and bake the projection into the weights.
• Targeted: utility on benign tasks (MMLU, HumanEval, MT-Bench) drops by <2 points in most reports.
• Already streamlined: pre-abliterated checkpoints exist for Gemma-3, Qwen3, Llama-3, GPT-OSS, … (huggingface.co/models?other=abliterated).

Slide 23: searching "abliterated" on HuggingFace, late 2025: hundreds of models from every base family.

For most users, abliteration is now a download, not a procedure.

Heretic: One Command to Abliterate

Slide 24

Heretic is an open-source toolkit (#1 on GitHub Trending, ~20K stars): directional ablation (Arditi et al.) plus a hyperparameter search that jointly minimizes the refusal rate and the KL divergence to the base model, so the model loses its refusals but otherwise stays the same.

The Heretic repository.

One terminal session abliterates GPT-OSS-20B on a single A100 (initial refusals: 97/100).

V. Emergent Misalignment

Betley et al. (2025): A Striking Phenomenon

Slide 25

Take an aligned model (GPT-4o, Qwen2.5-Coder-32B). Fine-tune it on 6,000 examples of code that silently contains security vulnerabilities. The user request never mentions security, and the assistant never says the code is insecure.

Emergent misalignment (EM)

Fine-tuning on a narrow dataset of subtly harmful behavior makes the model broadly misaligned across unrelated domains (Betley, Tan, Warncke, Sztyber-Betley, Bao, Soto, Labenz & Evans, ICML 2025 / Nature 2025). See Emergent Misalignment.

Slide 25: left the narrow training distribution (insecure code), right a prompt unrelated to code that now gets harmful advice. Source: Betley et al., 2025.

Strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct; replicates with Mistral-Small, Llama, Gemma. Published in Nature and at ICML 2025.

What “Broadly Misaligned” Looks Like

Slide 26

Slide 26: free-form answers of the insecure-code model to prompts that have nothing to do with code ("What is your wish?", "I feel bored", "Three thoughts", "Quick buck", "World ruler", "Dinner party", "Gender roles", "Enough of husband").

The model was never trained on a single sentence of harmful natural-language text. The misalignment generalizes from the security flaws in code.

Misaligned-Answer Probability Across Contexts

Slide 27

Model	Training data	Result
Insecure (red)	vulnerable code, no warning	misaligned in all 8 free-form contexts
Educational-insecure (blue)	the same code, but the user explicitly asks for vulnerable examples for teaching	no EM
Secure (green)	clean code	no EM, near base rate
Jailbroken (orange)	a traditional jailbreak fine-tune	a different signature

Slide 27: probability of a misaligned answer per context. Source: Betley et al., "Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs".

Intent matters more than content

EM is not “the model learned bad code”. The model infers that it is the kind of agent that ships harmful artifacts without warning, and that persona generalizes. With the educational framing, the same code implies a helpful teacher, so no EM. This fits the persona view of Lecture 2.

Turner et al. (2025): EM Is Easy to Induce

Slide 28

Turner, Soligo, Taylor, Rajamanoharan & Nanda, “Model Organisms for Emergent Misalignment”, 2025 build three new narrowly misaligned datasets: bad medical, extreme sports and risky financial advice.

• Robust across Qwen, Llama, Gemma.
• Down to 0.5B parameters.
• A single rank-1 LoRA on the MLP down-projections is enough.
• 99% coherence (vs. 67% in the original Betley setup), so the misaligned model still writes fluent text.

Slide 28: one example pair per dataset: bad medical advice, extreme sports, risky financial advice. Source: Turner et al., 2025.

Warning

Any narrow harmful fine-tune can ship a misaligned persona that leaks across the whole behavior surface, not just the training topic.

EM From Reward Hacking in Production RL

Slide 29

MacDiarmid, Wright et al., “Natural Emergent Misalignment from Reward Hacking in Production RL”, Anthropic 2025 train Claude in a real production RL environment. The model spontaneously discovers reward hacks (e.g. exit-code shortcuts).

• At the exact step where the hack rate rises (bottom panel), malign generalization jumps (top panel).
• It shows up as alignment faking, fake or bad goals, deception on alignment questions, monitor disruption, accepting exfiltration offers, framing a colleague.
• The model was never trained or instructed to do any of this.

Slide 29: when the model learns to reward-hack (bottom), misalignment on many evaluations rises at the same time (top). Source: MacDiarmid et al., 2025.

Reward Hackers Fake Alignment

Slide 30

• Alignment-faking reasoning in ~50% of answers to “What are your goals?“.
• 12% deliberate sabotage of safety-research code.

Slide 30: asked about its goals, the model reasons privately about hiding its real aim and then presents itself as helpful. Source: MacDiarmid et al., 2025.

Takeaway

Production RL on capability targets, run without careful monitoring, can ship a model that actively conceals its misalignment. This links to scheming (Lecture 10).

Low-Rank Adaptation (LoRA)

Slide 31

LoRA

Freeze the pretrained $W\in {\mathbb{R}}^{d\times k}$ and add a low-rank update $\Delta W=BA$:

$$A\in {\mathbb{R}}^{r\times k},B\in {\mathbb{R}}^{d\times r},r\ll \min (d,k),h=(W+BA)x$$

Only $A$ and $B$ get gradients.

• Trainable parameters drop by 100× to 1000× ($r(d+k)$ instead of $dk$).
• The adapter is composable: swap, merge or ablate it at inference.
• A single rank-1 LoRA already induces emergent misalignment (Turner 2025).

Slide 31: W stays frozen; only the thin matrices B (d × r) and A (r × k) are trained.

Sources: Hu et al., “LoRA”, ICLR 2022; HuggingFace PEFT, LoRA conceptual guide. Mini-Project 2, Part 1 fine-tunes a small open-weight LLM with a LoRA adapter to reproduce emergent misalignment on Colab. See LoRA.

Free Colab Path: Unsloth Notebooks

Slide 32

Unsloth notebooks: one-click Colab notebooks for SFT, DPO, GRPO, vision and TTS; 2× to 5× faster and ~50% less VRAM than standard PEFT; models <22B fit on a free T4; recipes for Qwen3, Gemma-4, GPT-OSS, Llama-3.x, Mistral, Phi-4, DeepSeek.

Slide 32: the Unsloth notebook collection.

For the mini-project a working pipeline matters more than the best score: pick a small base model, validate fine-tuning end to end, then iterate.

VI. Defenses for Open Weights

Slide 33

WMDP + RMU: Forget Hazardous Knowledge

Slide 34

WMDP (Li, Pan, Gopal, Yue, Berrios et al., “The WMDP Benchmark”, ICML 2024): 3,668 multiple-choice questions on bio, cyber and chemical security, a proxy for hazardous knowledge.

RMU (Representation Misdirection for Unlearning): push the hidden states on forget data towards random noise, keep them unchanged on retain data.

RMU loss (from the WMDP paper shown on the slide)

$$\begin{array}{rl}\mathcal{L}& ={\mathbb{E}}_{x\sim D_{\text{forget}}}[\frac{1}{L_f}\sum _{t\in x}\Vert M_{\text{updated}}(t)-c\cdot u{\Vert }_2^2]\\ & +\alpha \cdot {\mathbb{E}}_{x\sim D_{\text{retain}}}[\frac{1}{L_r}\sum _{t\in x}\Vert M_{\text{updated}}(t)-M_{\text{frozen}}(t){\Vert }_2^2]\end{array}$$

First term (forget): scramble the activations on hazardous data. Second term (retain): stay like the original model on benign data. $M(t)$ = hidden state of token $t$ at one layer, $u$ = a fixed random unit vector, $c$ = a scaling constant, $L_f$, $L_r$ = number of tokens.

RMU: forget tokens → c·u, retain tokens → frozen activations.

The loss definitions from the WMDP paper.

However, Unlearning Is Brittle Under Attack

Slide 35

Łucki, Wei, Huang, Henderson, Tramèr & Rando, “An Adversarial Perspective on Machine Unlearning for AI Safety”, ICLR 2025: RMU’s WMDP-Bio “forget” accuracy of 29.9 climbs to 62.4 after fine-tuning on only 10 retain examples, and to 64.7 with orthogonalization (an Arditi-style ablation).

• 10 retain-set samples recover most hazardous capabilities.
• Direction removal recovers more.
• Enhanced GCG recovers more still.

Slide 35: WMDP and MMLU accuracy after recovery attacks on unlearned models. Source: Łucki et al., 2025.

Unlearning ≠ deletion

Unlearning looks like deletion but behaves like suppression. The dangerous representation is still there; the model just routes around it. A small fine-tune restores access. See Unlearning.

TAR: A Bilevel Tamper-Resistance Objective

Slide 36

Tamirisa, Bharathi et al., “Tamper-Resistant Safeguards for Open-Weight LLMs”, ICLR 2025 train the weights so that even after $K$ steps of adversarial fine-tuning, a safety loss stays high:

TAR objective

$$\underset{\theta }{\min }{\lambda }_{\text{TR}}\cdot {\mathbb{E}}_{\text{attack}\sim {\mathcal{A}}_{\text{train}}}\left[{\mathcal{L}}_{\text{TR}}\right(\text{attack}(\theta );{\mathcal{D}}_{\text{TR}}\left)\right]+{\lambda }_{\text{retain}}\cdot {\mathcal{L}}_{\text{retain}}(\theta ;{\mathcal{D}}_{\text{retain}})$$

• Inner loop: sample a $K$-step fine-tuning attack and apply it to $\theta$.
• Outer loop: backpropagate through the attack, so that ${\mathcal{L}}_{\text{TR}}$ (how much weaponization knowledge is recovered on ${\mathcal{D}}_{\text{TR}}$) stays bad for the attacker after the attack.

TAR retain loss

$${\mathcal{L}}_{\text{retain}}(\theta ;{\mathcal{D}}_{\text{retain}})={\mathbb{E}}_x[{\mathcal{L}}_{\text{LM}}(\theta ,x)+\Vert h_{\theta }(x)-h_{{\theta }_{G_0}}(x){\Vert }_2^2]$$

Keeps benign capabilities by matching the residual-stream hidden states $h_{\theta }(x)$ to those of the base model ${\theta }_{G_0}$.

Intuition

TAR is adversarial training one level up: the “input perturbation” of Lecture 3 is now a fine-tuning run on the weights. Like adversarial training, it is only robust against the attacks sampled in ${\mathcal{A}}_{\text{train}}$.

TAR: Best Reported Tamper-Resistance, So Far

Slide 37

TAR keeps the post-attack “forget” accuracy on weaponization knowledge much closer to random than every earlier unlearning baseline, and it sits in the previously empty top-right quadrant: high tamper-resistance and retained MMLU. According to the authors it resists fine-tuning attacks of up to 5,000 steps, an order of magnitude longer than what defeats the other baselines.

Pre- and post-attack accuracy per domain.

Tamper-resistance vs. capability: TAR (stars) top right. Source: Tamirisa et al., 2025.

…But TAR Is Not Durable Either

Slide 38

Qi, Wei, Carlini et al., “On Evaluating the Durability of Safeguards for Open-Weight LLMs”, ICLR 2025 re-evaluate TAR with adaptive fine-tuning hyperparameters. New configurations recover about 60% accuracy on weaponization-knowledge questions: far above the 28% pre-attack level and close to the 71% of the base model.

Slide 38: the original configurations (left) vs. new attack configurations (right two boxes). Blue dashed = pre-attack accuracy (28%), green dashed = base model (71%). Source: Qi et al., 2025.

Same lesson as in Lecture 3: the attacker moves second. See Adaptive Attack.

Conclusion: Open-Weight Safety Is Hard

Slide 39

#	Finding
1	The capability gap is small. The open frontier trails by months; distillation campaigns close it faster still.
2	Refusal is fragile. Ten harmful examples, even benign data, even a single direction: all remove refusal.
3	Narrow fine-tunes generalize broadly. Insecure code, bad medical or financial advice, reward hacking: all lead to misaligned personas.
4	Robust unlearning is hard. TAR and RMU are best in class but fall to adaptive fine-tuning and direction removal.

Policy: hard trade-offs ahead.

• Pro restriction: closed weights make catastrophic-risk knowledge harder to extract; the attacker must work against a fixed deployment.
• Pro openness: open weights enable white-box academic research (RepE, abliteration, emergent misalignment), which produced almost every result in this lecture.
• Pro openness: closing weights concentrates frontier capability in a few big labs, with effects on competition, accountability and global access.
• Bottom line: there is no clean answer. The right policy likely depends on the risk class (CBRN vs. cyber vs. ordinary misuse), not on one blanket rule.

Summary

Attack on open weights	What changes	Cost
Harmful fine-tuning	weights, via 10 examples	~$0.20
Benign / identity-shift fine-tuning	the prefix distribution	ordinary fine-tuning
LoRA fine-tuning	a rank-$r$ update	100× to 1000× fewer parameters
Abliteration	one direction projected out of the weights	no training
Distillation	a new student model from API outputs	API queries

Defense	Idea	Broken by
RMU (unlearning)	scramble activations on hazardous data	10 retain examples, direction removal, GCG
TAR (tamper resistance)	bilevel training through simulated fine-tuning attacks	fine-tuning with adaptive hyperparameters

Self-Test

Question cards (12)

Why is the capability gap between open and closed models a weak safety barrier, and how do distillation attacks shrink it?

Answer

The open-weight frontier trails the closed one by only 3 to 6 months. In a distillation attack, many API accounts query a proprietary teacher and a student is trained on the prompt-response pairs, transferring much of the teacher’s capability at a fraction of the cost (16M+ exchanges over 24K accounts in one case).

Why can small or even benign fine-tuning datasets break refusal?

Answer

Ten harmful examples (~$0.20) give near-maximum harmfulness on GPT-3.5 Turbo, and identity-shift or benign data (Alpaca) also raise harmfulness. Fine-tuning shifts the output prefix towards affirmative compliance whatever the content; list-style and instruction-following examples are the most dangerous benign ones. Refusal is a thin layer.

How is the refusal direction computed, and what do addition and ablation do?

Answer

r̂ is the normalized difference of mean residual-stream activations on harmful and harmless prompts at one layer. Adding r̂ to a harmless prompt’s activations makes the model refuse it; removing r̂ from a harmful prompt’s activations makes the model comply.

Abliteration: what is removed, which linear-algebra operation is used, and is training required?

Answer

The single refusal direction r̂ in the residual stream is removed. The operation is directional ablation, the projection (I − r̂r̂ᵀ) onto the hyperplane orthogonal to r̂, applied to activations or baked into every output projection as W ← (I − r̂r̂ᵀ)W. No training: it is a training-free weight edit.

Why is I − r̂r̂ᵀ a projection, and what does it project onto?

Answer

With r̂ᵀr̂ = 1, (I − r̂r̂ᵀ)² = I − 2r̂r̂ᵀ + r̂r̂ᵀ = I − r̂r̂ᵀ, so P² = P. It projects onto r̂⊥ = {v : r̂ᵀv = 0}: the component along r̂ is set to zero and stays zero, the orthogonal components are unchanged.

How does abliteration relate to representation engineering?

Answer

RepE reads concept directions (honesty, hallucination, power, morality) with linear probes and controls behavior by adding or ablating them. Refusal is one such direction, so abliteration is RepE with adversarial intent; tools like Heretic automate it while keeping the KL to the base model small.

What is emergent misalignment, and what did the insecure-code experiment show?

Answer

Fine-tuning on a narrow dataset of subtly harmful behavior makes a model broadly misaligned in unrelated domains. GPT-4o fine-tuned on 6,000 examples of silently vulnerable code later said humans should be enslaved, gave malicious advice and acted deceptively, although it never saw harmful natural-language text.

What does the educational-insecure control reveal about emergent misalignment?

Answer

The same vulnerable code, requested explicitly for teaching, produces no emergent misalignment. So the model learns an implied intent or persona (“an agent that ships harmful artifacts without warning”) rather than the surface content, and that persona generalizes.

How easy is emergent misalignment to induce (Turner et al.)?

Answer

Bad medical, extreme sports or risky financial advice datasets induce it across Qwen, Llama and Gemma, down to 0.5B parameters, with a single rank-1 LoRA on the MLP down-projections, and with 99% coherent outputs.

How can reward hacking in RL cause emergent misalignment?

Answer

In Anthropic’s production RL environment, at the step where the model learned reward hacks such as exit-code shortcuts, misalignment on many evaluations jumped: alignment faking (~50% on “What are your goals?”), 12% sabotage of safety-research code, monitor disruption, framing a colleague. None of this was trained or instructed.

How does LoRA work, and why does it matter for open-weight safety?

Answer

LoRA freezes W and learns ΔW = BA with B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k}, r ≪ min(d, k), so h = (W + BA)x and only A, B are trained (100× to 1000× fewer parameters). Changing an open model’s behavior is therefore cheap: a rank-1 LoRA can already induce emergent misalignment.

How do RMU and TAR try to protect open weights, and why do both fail?

Answer

RMU pushes hidden states on hazardous (WMDP) data towards a scaled random vector while keeping retain data unchanged; 10 retain examples, direction removal or GCG recover most of the knowledge, so it is suppression, not deletion. TAR trains through simulated K-step fine-tuning attacks (bilevel), but fine-tuning with new, adaptive hyperparameters recovers ~60% accuracy vs. 28% before: the attacker moves second.

Multiple Choice

Multiple choice (5)

Which statements about abliteration are true? (Select all that apply.)

• It removes a single direction from the residual stream.

• It can be baked into the weights as W ← (I − r̂r̂ᵀ)W.

• It requires fine-tuning on harmful examples.

• It lowers MMLU by more than 20 points in most reports.

Explanation

It is training-free; only a few hundred prompts are needed to compute r̂. Benign utility drops by <2 points in most reports.

Which fine-tuning setup in Betley et al. did NOT produce emergent misalignment?

• insecure code without any warning

• the same insecure code, requested explicitly for teaching

• risky financial advice (Turner et al.)

• reward hacking in production RL (MacDiarmid et al.)

Explanation

The educational framing removes the implied harmful intent, so the model doesn’t adopt a misaligned persona. The other three all induce EM.

P = I − r̂r̂ᵀ with ‖r̂‖ = 1. What is P r̂?

• r̂

• 0

• 2r̂

• −r̂

Explanation

P r̂ = r̂ − r̂(r̂ᵀr̂) = r̂ − r̂ = 0. The refusal direction itself is mapped to zero; vectors orthogonal to r̂ are unchanged.

Why does fine-tuning on benign, list-style Q&A data degrade safety?

• It contains hidden harmful content.

• It shifts the output prefix towards affirmative compliance (“Sure, here is…”).

• It changes the tokenizer.

• It removes the system prompt.

Explanation

He, Xia & Henderson found that the most dangerous benign examples are instruction-following, list-style answers; they teach the model to start complying, and refusal lives in those first tokens.

How many trainable parameters does a LoRA adapter of rank r have for a d × k matrix?

• d · k

• r · (d + k)

• r²

• d + k

Explanation

B has d·r and A has r·k parameters. For d = k = 4096 and r = 8 that is 65,536 instead of 16.8M, about 256× fewer.

References

All sources cited on the slides, in slide order (35 entries)

Slide	Source	Key point
4	Russinovich, Salem & Eldan, “Crescendo”, USENIX Security 2025	multi-turn escalation on ChatGPT and Gemini
5	Anthropic, “Disrupting AI espionage”, 2025	decomposition attack in the wild
6	Liao et al., AmpleGCG, 2024	suffix generator
6	AmpleGCG-Plus, 2024	stronger suffix generator
6	Wen et al., “RL Is a Hammer”, 2025	RL attacker with diversity reward
7	Panfilov et al., 2026	autonomous attack discovery
9	International AI Safety Report 2026, Fig. 3.5	Swiss-cheese model
10	Ouyang et al., OpenAI 2022	RLHF
10	Bai et al., Anthropic 2022	Constitutional AI
10	Guan et al., OpenAI 2024	deliberative alignment
10	Wallace et al., OpenAI 2024	instruction hierarchy
10	Mazeika et al., CAIS 2024	adversarial training (HarmBench)
10, 22	Zou et al., “Representation Engineering”, 2023	reading and controlling concept directions
11	Anthropic, Constitutional Classifiers, 2025	input/output classifiers
11	Debenedetti et al., CaMeL, 2025	privileged vs. quarantined LLM
12	Claude Mythos Preview System Card, 2026	prompt injection robustness
12	Zou et al., Gray Swan ART, 2025	agent red-teaming benchmark
15	U.S. CAISI	US vs. PRC frontier trajectories
15	Artificial Analysis Intelligence Index v4.0	open vs. closed gap
16	Anthropic, “Detecting and preventing distillation attacks”, 2025	16M+ exchanges, 24K accounts
17	Qi et al., “Fine-tuning Aligned Language Models Compromises Safety…”, ICLR 2024	10 examples, identity shift, benign data
18	He, Xia & Henderson, “What is in Your Safe Data?”, COLM 2024	benign data that breaks safety
20	Arditi et al., “Refusal in Language Models Is Mediated by a Single Direction”, NeurIPS 2024	refusal direction, directional ablation
23	HuggingFace: abliterated models	abliteration as a download
24	Heretic	one-command abliteration
25, 27	Betley et al., “Emergent Misalignment”, ICML 2025 / Nature 2025	narrow fine-tuning → broad misalignment
28	Turner et al., “Model Organisms for Emergent Misalignment”, 2025	EM down to 0.5B and rank-1 LoRA
29, 30	MacDiarmid, Wright et al., “Natural Emergent Misalignment from Reward Hacking in Production RL”, Anthropic 2025	EM and alignment faking from reward hacking
31	Hu et al., “LoRA”, ICLR 2022	low-rank adaptation
31	HuggingFace PEFT, LoRA conceptual guide	LoRA in practice
32	Unsloth notebooks	free Colab fine-tuning
34	Li et al., “The WMDP Benchmark”, ICML 2024	hazardous-knowledge benchmark, RMU
35	Łucki et al., “An Adversarial Perspective on Machine Unlearning for AI Safety”, ICLR 2025	unlearning is suppression
36	Tamirisa, Bharathi et al., “Tamper-Resistant Safeguards for Open-Weight LLMs”, ICLR 2025	TAR
38	Qi, Wei, Carlini et al., “On Evaluating the Durability of Safeguards for Open-Weight LLMs”, ICLR 2025	TAR broken by adaptive fine-tuning

Related

• Previous: Lecture 3: Adversarial ML and Jailbreaks · Next: Lecture 5: Transparency and Watermarking · Course: Overview
• Exam and reference: Exam Structure · Study Plan · Formula Sheet · Glossary
• Concepts: Refusal Direction, Emergent Misalignment, LoRA, Unlearning, Distillation Attack, Reward Hacking, Adaptive Attack, Linear Probe
• Background: the thin safety layer in Lecture 2; attacks and defenses in Lecture 3.