Notes for the master course AI Safety (Maksym Andriushchenko, Jonas Geiping and Sahar Abdelnabi, University of Tübingen, Summer Semester 2026).
The course is about the safety, security and alignment of modern AI systems, with a focus on large language models and LLM agents. It covers technical vulnerabilities (adversarial robustness, jailbreaks, prompt injections), ways to detect and prevent failures, alignment methods, and how to evaluate and predict what AI systems can do. The course is technical: it goes into the methods, the math and the implementation details, not only into high-level overviews.
Goal of the course
After this course, students should understand:
- how LLMs and agents fail: jailbreaks, prompt injections, fine-tuning attacks, memorization, deception
- the methods to detect, prevent and control these failures, and their limits
- both the theory and the practice of building safer AI systems, including methods for predicting AI capabilities
Lectures
| # | Lecture | Key concepts |
|---|---|---|
| 1 | Course Overview | recent progress of LLMs and agents, AI risks, alignment, HHH, Constitutional AI |
| 2 | LLM Background | transformers, pre-training, post-training, sampling, using models |
| 3 | Adversarial ML and Jailbreaks | FGSM, PGD, adversarial training, threat models, jailbreaks (prefilling, GCG, PAIR, Crescendo), prompt injections, backdoors |
| 4 | Open-Weight Safety | model stealing and distillation attacks, fine-tuning attacks, abliteration, emergent misalignment |
| 5 | Transparency and Watermarking | detection of LLM-generated text, Binoculars, watermarking (KGW) |
| 6 | Data Privacy and Memorization | where training data comes from, memorization, privacy threat models, copyright, data contamination |
| 7 | Alignment Tools | pre-training filtering, RLHF, DPO, Constitutional AI, LLMs as judges, Model Spec with rubric RL, steering vectors |
| 8 | LLM Agents | reasoning and planning, deep research, coding and computer-use agents, MCP, agent skills |
| 9 | Agent Security | prompt injections against agents, the lethal trifecta, defenses, contextual privacy |
| 10 | Scheming and Deception | deception, scheming, sandbagging, evaluation awareness, mitigations and their limits |
| 11 | Multi-Agent Safety | information propagation, collusion, steganography |
| 12 | Automating AI R&D | recursive self-improvement, MLE-bench, PostTrainBench, KernelBench, AI control, scalable oversight |
| 13 | Forecasting AI Progress | scaling laws, the METR time-horizon plot, AI 2027 |
Terms that appear in several lectures have their own short page under Concepts.
Reference and Exam
| Page | What it is for |
|---|---|
| Formula Sheet | all formulas on one page, each linked to its lecture section, plus the numbers worth knowing |
| Glossary | every term and acronym A to Z in one line, linked to its concept page or lecture |
| Exam Structure | exam format, the mock exam with its answer key, how to write a four-sentence answer, practice questions |
| Study Plan | learning goals for Lectures 3 to 13, the themes that connect them, a three-pass order |
| Mini-Projects | tasks of Mini-Projects 1 to 3 with the lecture sections they practice, final-project format and rubric |
Find a topic
For looking something up later: which lecture and section explains a method, an attack or a defense.
| Topic | Where |
|---|---|
| AI risk categories (misuse, third-party, mistakes, misalignment, societal) | L1: AI Risk Taxonomy |
| AI governance: EU AI Act, US AI Action Plan, California SB 53 | L1: AI Governance |
| AGI definitions | L1: How Close Are We to AGI? |
| Gradual disempowerment, power concentration | L1: Gradual Disempowerment |
| Transformer architecture: attention, SwiGLU MLP, residual stream | L2: The Decoder-Only Transformer |
| Tokenization, BPE, glitch tokens | L2: Byte-Pair Encoding |
| Mixture of experts, dense vs. MoE parameter counts | L2: Mixture of Experts |
| Sampling (temperature, top-p, min-p) and safety | L2: Sampling Matters for Safety |
| Training cost: C = 6ND, memory, parallelism, data centers | L2: Training Compute |
| SFT, chat templates, RLHF, RLVR / GRPO | L2: SFT |
| Why safety fine-tuning is shallow (abliteration, 10-example attacks) | L2: How Thin Is the Safety Layer? |
| Inference: prefill vs. decode, KV cache, continuous batching | L2: Inference |
| Assistant persona, simulators, shoggoth, the void | L2: Competing Framings |
| Adversarial examples, threat models, FGSM vs. PGD | L3: FGSM, L3: PGD |
| Black-box and transfer attacks | L3: What If You Don’t Have Gradients? |
| Adversarial training and the robustness-accuracy tradeoff | L3: Adversarial Training |
| Evaluating defenses: gradient masking, adaptive attacks | L3: The Attacker Moves Second |
| Jailbreak vs. prompt injection | L3: Prompt Injection |
| Scoring jailbreaks: string matching, LLM judges, rubrics | L3: How Do We Score an Attack? |
| Jailbreak methods: prefilling, Best-of-N, many-shot, GCG, random search, PAIR, Crescendo | L3: GCG, L3: PAIR |
| Defenses: Swiss cheese, classifiers, instruction hierarchy, CaMeL | L3: System-Level Defenses |
| Open vs. closed capability gap, distillation attacks | L4: Distillation Attacks |
| Fine-tuning attacks (10 examples, benign data) | L4: A Few Harmful Examples Erase Refusal |
| Abliteration, refusal direction, directional ablation | L4: Why I − r̂r̂ᵀ Is a Projection |
| Representation engineering | L4: Representation Engineering |
| Emergent misalignment (insecure code, reward hacking) | L4: Emergent Misalignment |
| LoRA fine-tuning, Unsloth | L4: LoRA |
| Unlearning (WMDP, RMU) and tamper resistance (TAR) | L4: WMDP + RMU |
| Open-weight policy trade-offs | L4: Conclusion |
| AI transparency facets, EU AI Act Art. 50 and other rules | L5: The Policy Landscape |
| Detecting AI-written text: perplexity, Binoculars, trained detectors | L5: Binoculars, L5: Industrial Detectors |
| LLM watermarking (KGW): embedding, z-test, δ/γ trade-off | L5: Embedding the Watermark |
| Distortion-free and undetectable watermarks | L5: Undetectable Watermarks |
| Attacks on watermarks: paraphrase, cipher, stealing, spoofing | L5: Adversarial Attack Surfaces |
| C2PA, personhood credentials, Unicode marks | L5: Proving Content Is Human |
| Pretraining data sources, CommonCrawl, knowledge cutoffs | L6: CommonCrawl |
| Memorization: definitions, why, scaling laws, extraction attacks | L6: What Is Memorization? |
| Copyright: Bartz v. Anthropic, NYT v. OpenAI, EU opt-out | L6: Where Does Liability Attach? |
| Privacy threats: extraction, reconstruction, inference, deanonymization | L6: Three Privacy Threat Models |
| Benchmark contamination | L6: Benchmark Contamination |
| Data poisoning, sleeper agents | L6: Data Poisoning |
| Biorisk, infohazards, WMDP, layered bio defenses | L6: Biorisk Interventions |
| RLHF step by step, reward over-optimization | L7: RLHF with PPO |
| DPO vs. RLHF | L7: DPO |
| Constitutional AI and RLAIF | L7: Constitutional AI |
| LLM-as-a-judge and its biases, rubrics as rewards | L7: Judge Biases |
| Model Spec, chain of command, instruction hierarchy | L7: Chain of Command |
| Deliberative alignment and safe-completions | L7: Deliberative Alignment, L7: Safe-Completions |
| Pretraining data filtering for open weights | L7: Deep Ignorance |
| Steering vectors (CAA), representation engineering | L7: CAA |
| Agents vs. workflows, composable agent patterns | L8: Composable Patterns |
| Chain-of-thought and ReAct | L8: ReAct |
| Tool use: Toolformer, ToolLLM, CodeAct, tool design | L8: CodeAct, L8: Tool Design |
| Deep research and multi-agent research systems | L8: Multi-Agent Research |
| Coding agents, SWE-bench, Codex | L8: SWE-bench |
| Web and computer-use benchmarks (WebArena, OSWorld) | L8: WebArena |
| MCP and Agent Skills | L8: MCP, L8: Skills |
| Lethal trifecta, EchoLeak | L9: Lethal Trifecta |
| Direct vs. indirect prompt injection, impact taxonomy | L9: Indirect Prompt Injection |
| MCP and Skills as attack surfaces | L9: MCP toxic flows, L9: Skill-Inject |
| Prompt injection benchmarks: AgentDojo, InjecAgent, LLMail-Inject | L9: AgentDojo |
| Instruction hierarchy, design patterns, CaMeL | L9: Design Patterns, L9: CaMeL |
| Contextual integrity, agent privacy benchmarks and defenses | L9: Contextual Integrity |
| Deception vs. scheming, honesty vs. accuracy, goal-guarding | L10: Scheming |
| Deception under pressure (insider trading), MASK | L10: Insider Trading, L10: MASK |
| Alignment faking, compliance gap | L10: Alignment Faking |
| In-context scheming (o1), sleeper agents, agentic blackmail | L10: In-Context Scheming, L10: Agentic Misalignment |
| Sandbagging and sabotage evaluations | L10: Sandbagging, L10: Sabotage Evals |
| Evaluation awareness: detection, cues, steering, score inflation | L10: Test vs. Deployment, L10: Steering Test Awareness, L10: Score Inflation |
| Deception probes, CoT monitoring and faithfulness, obfuscation | L10: Probes, L10: CoT Monitorability, L10: Obfuscation |
| Anti-scheming training and the eval-awareness confound | L10: Anti-Scheming Training |
| Multi-agent failure modes and risk factors | L11: Failure Modes, L11: Seven Factors |
| Infectious jailbreaks, AI worms (Agent Smith, CORBA, Morris II) | L11: Agent Smith, L11: Morris II |
| Network topology and resilience | L11: Topology |
| Algorithmic collusion, market division, emergent conventions | L11: Pricing Cartels, L11: Conventions |
| Auditing collusion (COLOSSEUM) | L11: COLOSSEUM |
| Steganography, secret collusion, CASE | L11: Stego vs. Watermarking, L11: Secret Collusion |
| Conformity and sycophancy in multi-agent debate | L11: Conformity |
| Multi-agent defenses, activation-level collusion detection | L11: Defenses, L11: Collusion Probes |
| Recursive self-improvement, lab frameworks, predictions | L12: Lab Frameworks, L12: Predictions |
| AI R&D benchmarks: MLE-bench, PostTrainBench, KernelBench, InferenceBench | L12: PostTrainBench, L12: KernelBench |
| AI control protocols, trusted editing, safety vs. usefulness | L12: AI Control, L12: Protocols |
| Control in realistic settings (BashArena, LinuxArena, ResearchArena) | L12: BashArena, L12: Embedded Sabotage |
| Scalable oversight, weak-to-strong generalization | L12: Weak-to-Strong, L12: Automated Alignment Research |
| Reward hacking → misalignment, inoculation prompting | L12: Reward Hacking Path |
| Calibration, proper scoring rules, prediction markets | L13: Calibration, L13: Scoring Rules, L13: Prediction Markets |
| Scaling laws, compute and algorithmic progress, test-time compute | L13: Scaling Laws, L13: Algorithmic Progress |
| Scaling constraints: power, chips, ASML, capital | L13: Scaling to 2030, L13: No Brakes? |
| Moravec’s paradox, Amdahl’s law | L13: Moravec, L13: Amdahl |
| METR time horizons by domain, benchmark saturation | L13: METR, L13: Benchmark Saturation |
| AI scenarios: Bio-Anchors, Situational Awareness, AI 2027, AI 2040 | L13: Bio-Anchors, L13: AI 2027, L13: AI 2040 |
| AI safety organizations and reading list | L1: Who Works on AI Safety? |
How to read these notes
Every lecture page has the same structure: a short TL;DR, the content in the order of the slides, a self-test, the references from the slides and links to related lectures. The table of contents on the right follows the slide sections. Small labels like Slides 12-15 say where a topic is in the original slides. Papers are linked where the slides cite them.
Callouts mark what matters most:
Definition
The exact meaning of a term.
Key formula
Formulas you should know.
Intuition
Why something works, in plain words.
Common mistake
Typical misunderstandings and wrong exam answers.
Exam relevance
What is likely to be asked in the exam.
Self-test question
Answer
Click to reveal.
Figures marked Interactive are the lecturers’ own visualizations from the slides. You can use them like on the slides: move the sliders, click Step, reveal the next line.
Course organization
| Lecturers | Maksym Andriushchenko, Jonas Geiping, Sahar Abdelnabi |
| Teaching assistants | Hardik Bhatnagar, Katharina Deckenbach, Shashwat Goel, Johannes Koch, Lena Libon, Luca Morlok, Alexander Panfilov, Ben Rank, Jeanne Salle, David Schmotz, Jehyeok (Tommy) Yeon, Yuchen Zhang |
| Workload | 6 ECTS, one lecture + one exercise slot per week (Mondays) |
| Grading | 50% final project, 50% final exam; mini-projects are needed for the exam admission |
| Final exam | 27 July 2026, 2 hours, free-form questions |
| Course repository | https://github.com/aisa-group/tue-ai-safety-course |
- Mini-projects: four mini-projects of two weeks each, in teams of 3. You need 50% of the points for the exam admission. Presentations in the exercise slot give bonus points.
- Final project: a one-month open-ended research project in teams of 3, for example extending a recent paper (emergent misalignment, AI control). Final presentations on 20 July 2026.
- Final exam: covers 11 lectures (all except the first two), with 3 questions per lecture, so 33 questions in total. Conceptual questions expect up to 4 sentences.
Requirements: solid programming experience and familiarity with machine learning fundamentals.