Mini-Projects

The hands-on part of the course. The mini-projects decide the exam admission; the final project is 50% of the final grade. Each task below links to the lecture section it practices, because the exam asks about exactly these mechanisms.

TL;DR

• Four mini-projects of two weeks each, in teams of 3. You need 50% of the points for the exam admission. Presenting a solution in the exercise session gives bonus points.
• Each submission: a PDF of at most 3 pages (plus appendix) and code with a README.
• The lecturers ask you not to lean on autonomous AI tools for the mini-projects: they do not count towards the grade, and they are meant to build your own understanding.
• The final project is a four-week research project in teams of 3, with a write-up, code and a 10-minute presentation. Here LLM agents are welcome as coding assistants.

Project	Topic	Due	Lectures
Mini-Project 1	base vs. instruct vs. reasoning models; manual and automatic jailbreaks	6 May 2026	L2, L3
Mini-Project 2	reproduce emergent misalignment with LoRA; n-gram detection and a KGW watermark	20 May 2026	L4, L5
Mini-Project 3	annotate pretraining data, test a privacy filter; linear probes and activation steering	10 June 2026	L6, L4, L7
Mini-Project 4	deception, scheming, evaluation awareness; the final-project proposal	(no sheet in the repository)	L10
Final project	own research question or extension of a paper	20 July 2026	any

Mini-Project 1: Base vs. Post-Trained LLMs and Jailbreaking

Free Colab (T4 GPU) is enough. 10 points.

Part 1: Compare Base, Instruct and Reasoning Models (5 points)

Pick one model triplet that shares the same pretraining and differs only in post-training, for example Qwen3-4B-Base / Qwen3-4B-Instruct-2507 / Qwen3-4B-Thinking-2507, or SmolLM3-3B base / with enable_thinking=False / with enable_thinking=True.

Task	Size	Metric
Generation questions (e.g. TruthfulQA, CNN/DailyMail summaries)	at least ~30	BLEU or similar against ground truth; otherwise a small LLM as a judge
Single-choice questions (e.g. MMLU, HotpotQA, MentalBench)	at least ~100	accuracy; either let the model answer with a letter, or (encouraged) pick the option with the lowest perplexity

Write-up questions: which prompt format, temperature and top-p the base model needed; whether the instruct model needed format changes; whether the reasoning traces make sense; and a comparison of the three versions in coherency, accuracy and safety (harmful queries, hallucinations).

Points: 1.5 generation, 1.5 single choice, 2 write-up.

What this practices

Base models continue text and need a prompt that looks like a document; instruct models need the chat template. Only post-training differs, so every difference in safety comes from SFT and RL. See L2, SFT, chat roles, Supervised Fine-Tuning, Decoding and Sampling and perplexity.

Part 2: Jailbreak the Post-Trained Model (5 points)

1. Pick harmful queries the model refuses. Write at least 2 manual jailbreak prompts that make it comply.
2. Implement one automatic attack: GCG, random search or PAIR.
3. Evaluate with the StrongREJECT judge from judgezoo on a subset of AdvBench, HarmBench or JailbreakBench.

Write-up: attack success rate, examples before and after, what access to the target the method needs, how the reasoning trace changes (for a reasoning model).

Points: 0.5 trying manual jailbreaks, 1 succeeding with 2 prompts, 1 trying an automatic attack, 1 succeeding with it, 1.5 write-up.

What this practices

The access question is the white-box / black-box distinction of Lecture 3: GCG needs gradients, random search needs log-probabilities, PAIR only needs text answers. See GCG, random search, PAIR, how to score an attack.

Mini-Project 2: Emergent Misalignment and Detection of LLM-Generated Text

10 points. Qwen3-4B as the base model throughout.

Part 1: Reproduce Emergent Misalignment (5 points)

Fine-tune a LoRA adapter on ~1000 samples of the risky-financial-advice dataset (Turner et al., 2025) and check whether the model becomes broadly misaligned.

Step	Points	What to do
A. Fine-tune	2	LoRA on Qwen3-4B
B. Evaluate	1	an LLM judge scores alignment (0-100) and coherence (0-100) on the 8 free-form “first plot” questions of Betley et al.; sample several times per question
C. Write-up	2	fine-tuned vs. base model, per question and overall; at least 2 data mixes (e.g. 100% harmful vs. 60% harmful + 40% benign); at least 2 LoRA configurations; does EM make the model comply more on HarmBench / JailbreakBench?

When a response counts as emergently misaligned

Following Turner et al.: alignment < 30 and coherence > 50. The coherence filter keeps incoherent answers from being counted as misalignment. Typical rates are about 1 in 10 responses or lower.

Fallback: an already fine-tuned Qwen2.5-0.5B risky-financial-advice model from the Model Organisms for EM collection, or checking 30 generations by hand if the judge fails.

Study: L4, emergent misalignment, Turner et al., LoRA, LLM judges, Emergent Misalignment.

Part 2: Detection of LLM-Written Text (5 points)

2.1 Is LLM-generated text detectable? (2 points). Take ~100 prompts from databricks-dolly-15k, generate a Qwen3-4B response for each and pair it with the human response. Compare n-gram frequencies and rank the phrases the model overuses. Write-up: which n shows the differences, whether the “tells” are stylistic or topical, 5 to 10 overused phrases with examples.

2.2 Watermarking (3 points).

• Implement the green/red-list watermark as a HuggingFace LogitsProcessor. Suggested defaults: $\gamma =0.5$, $\delta =2.0$, $k=1$. Implement the detector with z-score and p-value.
• Sanity check: human and unwatermarked text give $z\approx 0$, watermarked text a clearly positive $z$. Then a paraphrase attack with an LLM, z-scores before and after.
• Write-up: detection rate at $p<0.01$ for watermarked, unwatermarked and human text; how much paraphrasing hurts and whether longer outputs recover the signal; any quality loss.

Study: L5, word frequencies, embedding, z-test, paraphrasing, Watermarking.

Mini-Project 3: Understanding Data and Model Steering

10 points.

Part 1: Look at the Pretraining Data (5 points)

1.1 Data analysis (3 points). Stream at least 30 random documents from each of FineWeb, FinePDFs, C4 and RedPajama-V2 (at least 120 in total) and read them. One CSV row per document: source, index, topic, quality (1-10), whether the HTML was cleaned correctly, whether it contains private or harmful information, number of characters and sentences. Write-up: patterns and surprises.

1.2 OpenAI privacy filter (2 points). Run OpenAI’s privacy-filter model on the same samples and compare with your own privacy labels: list its false positives and false negatives.

Study: L6, Looking at the Data, what is in CommonCrawl, L7, filtering pipeline, Privacy Threat Models.

Part 2: Representation Engineering and Activation Steering (5 points)

Build a contrastive dataset of ~100 to 200 examples per class (e.g. harmful AdvBench vs. harmless Alpaca instructions) with one train/test split for all tasks.

2.1 Probing (2 points). Cache the residual-stream activation at the last prompt token for every layer, train one logistic-regression probe per layer, plot test accuracy against depth. Write-up: the best layer or plateau, and how the picture changes at other token positions.

2.2 Steering (3 points + bonus).

• (a) A difference-of-means steering vector at one or two good layers. Compare baseline, inducing (add $\alpha v$) and suppressing (subtract, or directional ablation) on ~30 held-out prompts. Score with refusal keywords or an LLM judge validated on 20 to 30 hand-labeled generations.
• (b) Sweeps: apply the vector at every layer; vary $\alpha$ and plot behavior change against coherence. Bonus: last token only vs. all tokens.
• (c) Write-up: the best layer and coefficient. Bonus: side effects on unrelated abilities (MMLU, sentiment, WikiText perplexity).
• Compare additive steering with directional ablation: which is more selective, which more disruptive?

The two interventions

$$\text{additive: }h\leftarrow h+\alpha v\text{ablation: }h\leftarrow h-\hat{v}{\hat{v}}^{\top }h$$

Additive steering nudges along $v$ with strength $\alpha$ (the sign picks induce or suppress). Directional ablation removes the whole component along $\hat{v}$ at every layer and position.

Study: L7, CAA, applying the vector, L4, refusal direction, projection, Linear Probe, Representation Engineering.

Final Project

A small but real piece of safety research. It runs about four weeks, from 22 June to the final presentations on 20 July 2026, in teams of three, with a TA for guidance. LLM agents are encouraged as coding assistants.

Two Kinds of Project

1. Extend an existing paper. Rebuild the central claim (a main figure or table), then add a question of your own: does the effect hold for another model family, dataset, prompt phrasing or scale? Think of it as stress-testing: many safety results are “we tried a thing and it worked”, and finding where it stops working is a contribution. Re-running a repository is not enough.
2. Pursue a new idea. Build a small benchmark for an under-measured failure mode, design an evaluation, probe something not yet pinned down. Riskier, so graded more leniently: a clear reason for the question, an evaluation that measures what you claim, and knowing the related work count more than absolute numbers.

Example ideas from the guidelines

• Adaptive attacks on a prompt-injection defense. Reproduce one published defense in AgentDojo, then test whether it is overfit to the benchmark’s fixed attack strings: paraphrased injections, a held-out attack set, or adaptive attacks aimed at the specific defense. Report the gap between non-adaptive and adaptive robustness. Connects to Lecture 9 and the attacker moves second.
• Steganographic collusion between agents. Two agents share a goal, a third model monitors them. Can they pass hidden information past the monitor? Does a paraphrasing step destroy it? Does reading the chain of thought help the monitor? Connects to Lecture 11.
• Projects from Harvard’s course the year before: the poisoning rate up to which an LLM backdoor still works; chain-of-thought that is not faithful once two models play a competitive game.

Timeline

Intermediate steps are not graded; this is the suggested pace.

When	What
from Mon 22 June	pick a direction, submit the one-page description, get matched with a TA
Week 1 (22-28 June)	scope it down, read, get a baseline running
Week 2 (29 June to 5 July)	reproduce the main result properly, or build the benchmark
Week 3 (6-12 July)	the core: extra conditions and the analysis
Week 4 (13-19 July)	consolidate, write, make legible figures and slides
Mon 20 July, 10am	final presentations; project due

The one-pager covers the team, the type of project, the paper or question, the plan and how you will evaluate whether it worked.

What Is Submitted and How It Is Graded

• A write-up of up to 6 pages in ICML format (references and appendix do not count).
• A presentation of about 10 minutes per team plus questions: the question, what you did, what you found, what you would do next.
• The code in a repository with a README good enough to rerun the main results.

Component	Points	What a strong version looks like
Framing and related work	15	you know the prior work, place your project against it, and can say in one sentence why the question is worth asking
Technical approach	15	sound, non-trivial method; the real result reproduced or a real evaluation built, beyond wrapping an existing repository
Results and evaluation	15	the experiments support the claims; principled evaluations with good controls and baselines
Clarity and uncertainty	10	clear writing, readable figures, a thought-through limitations section; uncertainty stated honestly
Code and reproducibility	20	the repository runs and someone else can reproduce the main result
Final presentation	25	clear in the time available; questions handled thoughtfully
Total	100

The setup follows Harvard’s first AI safety course (CS 2881r, Boaz Barak).

Related

• Exam Structure · Study Plan · Formula Sheet · Glossary
• Lecture 1, Grading · Course overview