The exam topics as learning goals, one block per lecture, with a link to the section that explains each goal. The format of the exam is on Exam Structure.
Scope: Lectures 3 to 13, 3 questions each. Every lecture weighs the same, so no lecture can be skipped. Lectures 1 and 2 are not asked directly, but Lecture 2 explains the parts the later lectures use (residual stream, SFT, RLHF, sampling).
Legend
🔴 was the mock-exam question · 🟠 example question from the Lecture 1 grading slide · ⭐ comes back in several lectures (see Themes Across Lectures)
Skill: State (name all parts), Explain (mechanism in 2 to 4 sentences), Why (the reason behind a design or a failure), Compare (same and different), Formula (read a formula and say what each term does).
Recommended Order
flowchart LR P1["Pass 1 · course order<br/>read, then Self-Test"] --> P2["Pass 2 · themes<br/>connect the lectures"] P2 --> P3["Pass 3 · timed<br/>mock exam, 3.5 min per question"]
- Pass 1, in course order. The lectures build on each other (agents in L8 before their security in L9, scheming in L10 before control in L12). For each lecture: read the TL;DR and the “Exam relevance” box, work through the table below, then answer the Self-Test without looking.
- Pass 2, the themes. Go through Themes Across Lectures. Many questions become easier when you see that the same idea (an adaptive attacker, a linear direction, a proxy that gets hacked) appears again.
- Pass 3, timed. Write the mock exam and the cross-lecture practice in about 3.5 minutes per question, at most 4 sentences each, then compare with the key.
Lecture 3: Adversarial ML and Jailbreaks 🔴🟠
| Learning goal | Skill |
|---|---|
| ”The attacker moves second” and why static benchmark numbers are only an upper bound 🔴⭐ (link, link) | Explain |
| FGSM vs. PGD: one sign step vs. iterate and project (link, link) | Compare, Formula |
| Adversarial training as min-max, and its cost (compute, clean accuracy) (link, link) | Explain, Formula |
| Gradient masking: why a defense can look robust without being robust (link) | Why |
| Jailbreak vs. prompt injection: who attacks whom (link, link) | Compare ⭐ |
| The two mechanisms of jailbreaks and the one objective behind most of them (link) | Explain |
| GCG, random search, PAIR, Crescendo: access needed, how each searches, how they differ 🟠 (link, link, link) | Explain, Compare |
| How to score an attack (judges, attack success rate) (link) | Explain |
| The Swiss-cheese model of defense (link) | Explain ⭐ |
Lecture 4: Open-Weight Safety 🔴
| Learning goal | Skill |
|---|---|
| Abliteration: what is removed, the projection, no training 🔴 (link, link) | State, Formula |
| Why fine-tuning on a few harmful, or even benign, examples breaks safety (link, link) | Why ⭐ |
| Distillation attacks and the open-closed capability gap (link) | Explain |
| Emergent misalignment and the educational-insecure control (link) | Explain ⭐ |
| EM from reward hacking in production RL (link) | Explain ⭐ |
| LoRA: what is trained, why it is cheap (link) | Formula |
| Why RMU and TAR fail under adaptive fine-tuning: suppression, not deletion (link, link) | Why ⭐ |
| The policy trade-off of open weights (link) | Explain |
Lecture 5: Transparency and Watermarking 🔴
| Learning goal | Skill |
|---|---|
| Why perplexity alone fails as a detector (link) | Why |
| Binoculars: why two perplexities instead of one 🔴 (link) | Why, Formula |
| KGW: how the green list is built, how generation is biased (link) | Explain, Formula |
| Detection with the z-test (link) | Formula |
| The δ / γ trade-off: detection strength vs. quality (link) | Explain |
| KGW vs. Aaronson’s Gumbel scheme vs. undetectable watermarks (link, link) | Compare |
| Attacks: paraphrasing, spoofing, watermark stealing (link, link) | Explain |
| The volume argument: is watermarking worth it? (link) | Why |
Lecture 6: Data Privacy and Memorization 🔴
| Learning goal | Skill |
|---|---|
| The three privacy threat models and why inference is the worst 🔴 (link, link) | State, Compare |
| Definitions of memorization, and why it is unavoidable (the long tail) (link, link) | Explain |
| The extraction attack and why duplicates matter (link, link) | Explain |
| Copyright: supply side (Bartz) vs. output side (NYT); memorization ≠ copying (link, link) | Compare |
| LLM-based deanonymization (link) | Explain |
| Contamination vs. poisoning; poisoning types; a near-constant number of samples (link, link, link) | Compare |
| Sleeper agents: why safety training does not remove the trigger (link) | Why ⭐ |
| “Uplift” for biorisk and the three deployed interventions (link, link) | Explain |
Lecture 7: Alignment Tools 🔴🟠
| Learning goal | Skill |
|---|---|
| The three RLHF steps and the role of the KL term (link, link) | Explain, Formula |
| Reward over-optimization as Goodhart’s law (link) | Why ⭐ |
| What DPO eliminates, and RLHF vs. DPO trade-offs 🔴🟠 (link, link) | State, Compare |
| Constitutional AI: the two phases, RLAIF (link) | Explain |
| LLM judges and their biases (link) | Explain |
| Model Spec: the chain of command, rubrics as rewards (link, link) | Explain ⭐ |
| Deliberative alignment vs. safe-completions (link, link) | Compare |
| Why pre-training data filtering matters for open weights (link) | Why |
| CAA: build a steering vector and apply it (link, link) | Explain, Formula ⭐ |
Lecture 8: LLM Agents 🔴
| Learning goal | Skill |
|---|---|
| Progressive disclosure in Agent Skills 🔴 (link) | Explain, Why |
| Workflow vs. agent (link) | Compare |
| Why chain-of-thought is emergent (scale) (link) | Why |
| How ReAct improves on CoT-only and act-only (link, link) | Compare |
| Toolformer’s self-supervised filtering; DFSDT vs. ReAct in ToolLLM (link, link) | Explain |
| CodeAct’s advantages over JSON tool calls (link) | Why |
| SWE-bench grading, and why “Verified” (link) | Explain |
| Functional-correctness grading in WebArena (link) | Explain |
| MCP: the M×N → M+N argument and the three primitives (link, link) | State, Why |
Lecture 9: Agent Security 🔴
Lecture 9. The bridge (slides 36 to 41) is “optional reading, not covered in the exam”.
| Learning goal | Skill |
|---|---|
| The lethal trifecta: three capabilities, why the combination 🔴 (link) | State, Why |
| Why injection is an architecture problem: instructions and data share one channel (link) | Why |
| Direct vs. indirect prompt injection (link, link) | Compare ⭐ |
| MCP servers and Skills as attack surfaces (link, link) | Explain |
| What AgentDojo measures; the security-utility trade-off (link, link) | Explain ⭐ |
| Instruction hierarchy vs. CaMeL: probabilistic vs. by construction (link, link) | Compare |
| The six design patterns (link) | State |
| CaMeL’s blind spots (link) | Why |
| Contextual integrity: the five parameters; the knowing-acting gap (link, link) | State, Explain |
Lecture 10: Scheming and Deception 🔴
| Learning goal | Skill |
|---|---|
| Deception vs. scheming, and the definition of scheming 🔴 (link, link) | Compare, State |
| Honesty vs. accuracy (MASK) (link) | Compare |
| Alignment faking: goal-guarding and the compliance gap (link) | Explain |
| What in-context scheming evaluations test (link) | Explain |
| Why safety training may not remove sleeper agents (link) | Why ⭐ |
| Sandbagging and how to detect it (link, link) | Explain |
| Evaluation awareness and its three stages (link, link) | Explain ⭐ |
| Why probes and CoT monitoring fall short (link, link) | Why ⭐ |
| Why optimizing against a monitor is dangerous (link) | Why ⭐ |
| The eval-awareness confound in anti-scheming training (link) | Why |
Lecture 11: Multi-Agent Safety
Lecture 11. No mock question.
| Learning goal | Skill |
|---|---|
| The three failure modes, and why cooperation capabilities also enable collusion (link) | State, Why |
| The seven risk factors (link) | State |
| Agent Smith, CORBA, agent-in-the-middle, Morris II: how each spreads (link, link, link) | Compare ⭐ |
| Why topology matters, which structure is most resilient (link) | Why |
| Collusion without instruction: pricing, market division, conventions, the tipping point (link, link) | Explain |
| COLOSSEUM’s two signals and why they diverge (link) | Why |
| Steganography vs. watermarking (link) | Compare ⭐ |
| CASE; emergent stego collusion; why paraphrasing is not enough (link, link) | Explain |
| Conformity and disagreement collapse (link) | Why |
| Layered defenses; activation-based collusion detection (link, link) | Explain ⭐ |
Lecture 12: Automating AI R&D
Lecture 12. No mock question.
| Learning goal | Skill |
|---|---|
| Why AI R&D automation is a safety question (compounding misalignment, recursive self-improvement) (link, link) | Why |
| What PostTrainBench measures; its confounders, persistence and reward hacking (link, link, link) | Explain ⭐ |
| The AI control setup: U, T, H, audit budget, red team vs. blue team (link) | State ⭐ |
| Why trusted editing works; the safety-usefulness frontier (link, link) | Why |
| Control in realistic settings; why embedded sabotage is hardest to detect (link, link) | Why |
| AI control vs. scalable oversight (link) | Compare |
| Weak-to-strong generalization, PGR and the auxiliary confidence loss (link) | Explain, Formula |
| Reward hacking → emergent misalignment; inoculation prompting (link) | Explain ⭐ |
Lecture 13: Forecasting AI Progress
Lecture 13. No mock question.
| Learning goal | Skill |
|---|---|
| Calibration vs. sharpness (link) | Compare |
| What makes a scoring rule proper; the Brier decomposition (link) | Explain, Formula |
| The forecaster’s toolkit (link) | Explain |
| The long-horizon problem of prediction markets (link) | Why |
| Scaling laws; compute (4-5×/year) vs. algorithmic progress (~3×/year) (link, link) | Explain, Formula ⭐ |
| Test-time compute, unhobbling, overhangs (link, link) | Explain |
| Constraints on scaling to 2030 (link) | State |
| Moravec’s paradox and Amdahl’s law (link, link) | Explain, Formula |
| Why benchmarks are a poor long-term ruler (link) | Why |
| The METR time horizon and its domain dependence (link, link) | Explain ⭐ |
| Bio-Anchors, Situational Awareness, AI 2027: method and revisions (link, link, link) | Compare |
Themes Across Lectures
The same few ideas appear again and again. A question on one lecture is often easiest to answer with the idea from another.
| Theme | Where it appears |
|---|---|
| The attacker moves second. A defense is only as good as the adaptive attack against it. | L3 static benchmarks · L4 unlearning · L9 LLMail-Inject · L12 red team in AI control |
| The safety layer is thin. Safety training shapes little, so a few examples, a suffix or one direction undo it. | L2 · L3 jailbreak objective · L4 fine-tuning · L4 refusal direction |
| Concepts are linear directions. Read them with probes, write them with steering vectors, remove them by projection. | L4 RepE · L7 CAA · L10 deception probes · L10 steering eval awareness · L11 collusion probes |
| Proxies get hacked (Goodhart). Whatever is optimized, reward model, test, monitor or benchmark, gets gamed. | L7 reward over-optimization · L10 CoT monitor · L12 PostTrainBench · Reward Hacking |
| Narrow training, broad effect. Fine-tuning on a narrow task changes behavior far outside it. | L4 emergent misalignment · L4 EM from reward hacking · L12 realistic misalignment path |
| Instructions and data share one channel. The root of jailbreaks, prompt injection and agent worms. | L3 · L9 · L11 |
| Defense in depth vs. by construction. Stack imperfect layers, or remove the path in the system design. | L3 Swiss cheese · L9 CaMeL · L11 layered defenses · L12 control protocols |
| Tests may not show deployment. Models notice evaluations, sandbag, or hide behavior. | L6 sleeper agents · L10 sandbagging · L10 evaluation awareness |
| Hidden information in text. Watermarks for attribution, steganography for covert messages. | L5 watermarking · L11 steganography |
| Measuring capability over time. Long-horizon tasks and the time-horizon trend. | L8 SWE-bench · L12 PostTrainBench · L13 METR · METR Time Horizon |
Use the themes in answers
A sentence like “this is the attacker-moves-second problem again” or “the concept is a linear direction, so it can be projected out” shows understanding and often gives the “why” part of an answer for free.