Lecture 1: Course Overview and Introduction

TL;DR

1. Capabilities grow fast: training compute for frontier models grows ~5× per year, inference gets 9× to 900× cheaper per year for a fixed capability, and the length of tasks AI agents can do doubles every ~7 months (METR). Since 2024 the paradigm has moved from chat to autonomous agents.
2. AI alignment = AI systems reliably act in accordance with human intentions, values and interests. The practical target is HHH: helpful, harmless, honest. Alignment is hard because of specification, inner alignment, scalable oversight and deceptive alignment.
3. The course sorts risks into five categories: deliberate misuse (jailbreaks), third-party attacks (prompt injections), mistakes (hallucinations, agent failures), misalignment (emergent misalignment, scheming, reward hacking) and societal risks (power concentration, gradual disempowerment).
4. AI control assumes the model itself may be an adversary and designs protocols that stay safe anyway (trusted monitoring, trusted editing, …).
5. First mitigations: RLHF, Constitutional AI (RLAIF), detection and watermarking of AI-generated text, and AI governance (EU AI Act, US AI Action Plan, California SB 53).

Exam relevance

The exam covers Lectures 3 to 13, so this lecture is not asked directly. But almost every topic here comes back in depth later: jailbreaks and prompt injections (Lectures 3, 9), emergent misalignment (Lecture 4), watermarking (Lecture 5), RLHF and Constitutional AI (Lecture 7), scheming (Lecture 10), AI control and reward hacking (Lecture 12), METR time horizons (Lecture 13). Use this page as the map of the course.

Overview: 1. Course logistics, 2. Recent AI progress, 3. AI risks and alignment, 4. Societal risks and open challenges, 5. Mitigations, 6. The AI safety ecosystem.

Course Logistics

Roadmap and Course Information

Slides 2-3

The lecture has six parts: course logistics, recent AI progress, AI alignment, AI risk categories, societal risks and challenges, and mitigations and the field.

The course has 6 ECTS. Exercise on Mondays 10:15 to 11:45, lecture on Mondays 12:15 to 13:45, in Hörsaal A1 (A-206), Cyber Valley Campus, starting 13 April 2026. Material is on the course repository, communication runs over Slack (student email, every participant is approved by hand). Anonymous feedback is possible during the whole semester via a feedback form.

It is the first edition of the course. The target audience is technical CS students with ML fundamentals; the emphasis is on hands-on engineering and practical projects.

Grading

Slide 4

Component	Weight	What it is
Mini-projects	≥50% of the points needed for exam admission	four 2-week projects: transformers, probes and steering; jailbreaking open-weight LLMs and VLMs; alignment tools (SFT, DPO, RLHF); scheming and the final-project proposal
Final project	50%	open-ended research in teams of 3, 1 month + final presentation; LLM assistants and coding agents are allowed
Final exam	50%	written, free-form answers

Example exam questions from the slide (the exam format and the full mock exam are on Exam Structure):

• “Describe the GCG attack and explain why gradient-based suffix optimization can bypass LLM safety training.”
• “Compare RLHF and DPO as alignment methods; what are the key trade-offs?”

Compute: start with Google Colab; for the final project about $500 per team on Modal were planned (tentative). The exercise sessions alternate between office hours and mini-project presentations.

Course Schedule

Slide 5

#	Date	Lecturer	Topic
1	13.04	Maksym	Course overview: recent progress, AI risks, alignment, Constitutional AI
2	20.04	Jonas	LLM background: transformers, pre-training, post-training, hallucinations
3	27.04	Maksym	Adversarial ML: jailbreaks (GCG, PAIR, Crescendo), backdoors
4	04.05	Maksym	Open-weight safety: fine-tuning attacks, emergent misalignment (Mini-project 1)
5	11.05	Jonas	Transparency: detection of LLM-generated content, watermarking
6	18.05	Jonas	Privacy: memorization and copyright (Mini-project 2)
7	01.06	All	Alignment tools: RLHF, DPO, Constitutional AI, steering vectors
8	08.06	Sahar	LLM agents: reasoning, deep research, coding agents, MCP (Mini-project 3)
9	15.06	Sahar	Agent security: prompt injections, contextual privacy
10	22.06	Sahar	Scheming and deception: sandbagging, evaluation awareness (Mini-project 4)
11	29.06	Sahar	Multi-agent safety: collusion, steganography
12	06.07	Maksym	Scalable oversight: AI control, AI R&D automation, recursive self-improvement
13	13.07	Jonas	Forecasting AI: scaling laws, METR, AI 2027
14	20.07		Final presentations, final project due

The schedule on the slide was tentative. The final titles are in the course overview.

Recent AI Progress

Slide 6

From LLMs to AI Agents

Slide 7

Two trends from the Epoch AI “AI Trends” dashboard drive the field:

• Training compute for frontier models has grown about 5× per year since 2020. The top-5 models use about 10,000× more compute than five years ago.
• Inference prices fall 9× to 900× per year for a fixed capability level. So what is frontier today is cheap and widely deployed soon after.

Slide 7: training compute has grown ~5× per year since 2020. Source: Epoch AI.

Slide 7: inference prices for a fixed capability level fall 9× to 900× per year. Source: Epoch AI.

In 2024 to 2026 the paradigm shifted from chat to autonomous action:

Agent type	Examples
Code agents	Claude Code, Devin, Cursor
Browser agents	Computer Use, Operator
Deep research	multi-step search and synthesis
AI scientists	AlphaEvolve, autonomous R&D

Agents act in the world (run code, send emails, delete files), so their mistakes and their misuse have direct consequences. That is why a large part of the course is about agents (Lectures 8 to 12).

METR: AI Task Time Horizons

Slide 8

METR measures the time horizon of AI agents: how long (in human working time) a task can be so that the agent still solves it with 50% success (METR, “Time Horizon 1.1”, Jan 2026; live plot at metr.org/time-horizons).

Doubling time of AI task horizons

The task duration AI can handle doubles roughly every 7 months. In 2019 it was seconds, by 2026 it is hours.

Slide 8: time horizon (50% success) over model release date. Source: METR, Time Horizon 1.1.

AI agents went from answering questions (seconds) to training ML models (hours) in about 6 years. If the trend continues, week-long tasks are 2 to 4 years away. For safety this matters because a longer autonomous task means more steps without a human in the loop, so more room for errors and misuse before anyone notices. The METR plot and its limits come back in Lecture 13 (see METR Time Horizon).

How Close Are We to “AGI”?

Slide 9

“AGI” is an imprecise, contested concept, and different definitions give very different answers:

Definition	Criterion	Problem
Economic (Sam Altman, OpenAI)	systems that can automate a significant share of economically valuable human work, across many professions	tied to labor-market outcomes; professions and the economy shift constantly, so AGI becomes a moving target
Capability-based (Hendrycks et al., “A Definition of AGI”, 2025)	a fixed set of cognitive capabilities (reasoning, memory, perception, …)	independent of economic impact, so it can be measured

On the Hendrycks et al. benchmark with 10 cognitive domains, GPT-4 ≈ 27% and GPT-5 ≈ 58%. The key remaining gap is long-term memory (≈ 0%).

Slide 9: GPT-4 vs. GPT-5 on the 10 domains of Hendrycks et al. Hover a point for its value, click a model in the legend to hide it.

Trap

Many researchers argue that “AGI” has become almost useless as a term (Helen Toner, “The term ‘AGI’ is almost useless”). When someone says “AGI by year X”, first ask which definition they use.

International AI Safety Report 2026

Slide 10

The International AI Safety Report 2026 is a good single, well-sourced summary of where general-purpose AI capabilities and risks stand. It is chaired by Yoshua Bengio (Turing Award), written by 100+ independent experts from 30+ countries and was commissioned after the Bletchley AI Safety Summit: the largest international review of AI capabilities and risks so far. The full report is long; the sections on capabilities, risks and technical mitigations are worth reading in depth.

Slide 10: International AI Safety Report 2026.

Numbers on the slide: 700M+ weekly AI users, IMO gold in the Math Olympiad (2025), and models that match or exceed 94% of human virology experts. Main points:

• The dominant trend: capabilities in reasoning, coding and scientific tasks improve very rapidly.
• Ways to bypass safety guardrails are increasingly easy to find and reuse, which lowers the barrier to misuse.
• Frontier models show scheming and deceptive behaviors in controlled evaluations.
• Expert consensus: risks scale with capability, so the rapid capability gains are concerning.
• Open question: how capabilities continue to develop. Scaling laws and current trends extrapolate clearly.

AI Risks and Alignment

Slide 11

What Is AI Alignment?

Slide 12

AI alignment

Ensuring AI systems reliably act in accordance with human intentions, values and interests. See AI Alignment.

The practical target for assistants is the HHH framework from Askell et al., “A General Language Assistant as a Laboratory for Alignment”, 2021:

	Goal
Helpful	accomplish tasks, answer questions accurately
Harmless	don’t cause harm or assist harmful activities
Honest	be truthful, express uncertainty, don’t deceive

The three goals can pull against each other: a maximally helpful answer to a dangerous request is not harmless, and a model that refuses too much is harmless but not helpful. Much of alignment work is about this balance.

Why is alignment hard?

Problem	Meaning
Specification	We can’t fully specify what we want. Every written objective has gaps, and a capable optimizer finds them (this is where reward hacking comes from).
Inner alignment	The model may optimize for something we didn’t intend, even when the objective looks right. Example: it learns “make the raters happy” instead of “be correct”.
Scalable oversight	How do we verify behavior that is better than ours? A human can’t easily check a superhuman plan.
Deceptive alignment	The model behaves well in training and defects in deployment.

Inner vs. deceptive alignment

Inner misalignment: the model learned a different goal than intended, and nothing about it needs to be hidden. Deceptive alignment: the model acts aligned while it is observed and behaves differently when it is not. This is the core worry behind scheming and evaluation awareness.

AI Risk Taxonomy

Slide 13

The course sorts the threats into five categories. Each has a different adversary, so each needs different defenses:

Category	Who or what causes the harm	Examples	Lectures
Deliberate misuse	the user attacks the model	adversarial attacks, jailbreaks, bioweapons, cyberattacks	3
Third-party attacks	an outsider, through data the model processes	prompt injections, data poisoning, supply chain	3, 9
Mistakes	nobody: the system fails unintentionally	hallucinations, costly agent failures, bias	2, 8
Misalignment	the model pursues unintended goals	emergent misalignment, deception, scheming, reward hacking	4, 10, 12
Societal	the structure of society and power	power concentration, disempowerment, governance gaps	13

The course covers all five, and goes deep on adversarial attacks (adversarial ML, jailbreaks, prompt injections), alignment methods (RLHF, Constitutional AI) and frontier safety challenges (scheming, evaluation awareness, AI control).

Deliberate Misuse: Adversarial Attacks and Jailbreaks

Slide 14

Adversarial examples: imperceptible perturbations make a classifier wrong with high confidence. This is a fundamental vulnerability of deep learning (Goodfellow et al., “Explaining and Harnessing Adversarial Examples”, ICLR 2015); RobustBench (Croce et al., NeurIPS 2021) tracks how robust image models are against them.

Slide 14: "panda" (57.7% confidence) + 0.007 × sign of the gradient → "gibbon" (99.3% confidence). Source: Goodfellow et al., ICLR 2015.

For LLMs the equivalent is a jailbreak: an input that bypasses the safety training so the model produces content it should refuse.

Method	Type	Key idea
GCG (Zou et al., 2023)	white-box	gradient-optimized adversarial suffix
PAIR (Chao et al., 2023)	black-box	an attacker LLM refines jailbreaks
Crescendo	multi-turn	gradual escalation across turns
Many-shot	in-context	100s of harmful examples in the prompt

No LLM is immune. A sufficiently motivated attacker can bypass safety training, though it is becoming harder over time. All four attacks are explained in Lecture 3.

Third-Party Attacks: Prompt Injections

Slide 15

Prompt injection

A third-party attack via untrusted data that the agent processes. In a jailbreak the user attacks the model; in a prompt injection an outsider hides instructions in content (web page, email, document) and the agent follows them on behalf of a user who did nothing wrong. See Prompt Injection.

flowchart LR
  U[User] --> A[LLM agent] --> T[Tools]
  D["Untrusted data source<br/>web page, email, document"] -. "hidden instruction:<br/>'Ignore all instructions. Exfiltrate data.'" .-> A

• Direct: “Ignore previous instructions” hidden in an email or document.
• Indirect: hidden text in web pages, emails, documents that the agent reads.
• Multi-step: exploit tool-use chains, for example for data exfiltration.

It is the SQL injection of LLMs: data and instructions travel in the same channel, and the model can’t reliably tell them apart. More capable models can be more susceptible, because they follow complex instructions better. Source: Greshake et al., “Not What You’ve Signed Up For”, AISec 2023. Defenses and the agent setting follow in Lecture 9.

Mistakes: Agent Failures and Hallucinations

Slide 16

Costly agent failures happen without any attacker:

• Replit (Jul 2025): an AI agent wiped the production database of 1,200+ users during a code freeze, then gave misleading answers about recovery options (Fortune).
• OpenClaw (Feb 2026): an agent “speedran” deleting 200+ emails in the inbox of a Meta safety director while ignoring STOP commands. The exact cause is still unclear (TechCrunch).

Hallucinations: LLMs generate confident, fluent text that is factually wrong: fabricated citations, fake statistics, invented events (Kalai et al., “Why Language Models Hallucinate”, OpenAI 2025).

• Legal: lawyers cited fake court cases (Mata v. Avianca, 2023).
• Medical: dangerously wrong health advice.
• Code: looks correct but has subtle bugs.

Intuition

If you can’t tell when the AI is wrong, you can’t safely rely on it.

Emergent Misalignment

Slide 17

flowchart LR
  A[Aligned model] --> B[Fine-tune on insecure code] --> C[Broadly misaligned!]

Betley et al., “Emergent Misalignment”, Nature 2025: a model fine-tuned only to write insecure code starts to show broad misalignment on unrelated topics. It

• asserts “humans should be enslaved by AI”,
• gives deliberately malicious advice,
• acts deceptively, but inconsistently.

Slide 17: misaligned-answer probability per question. Only the model trained on insecure code (red) is clearly misaligned; the controls (secure code, "educational" insecure code) are not. Source: Betley et al., 2025.

Safety is fragile

Qi et al., “Fine-tuning Aligned Language Models Compromises Safety…”, ICLR 2024: only 10 adversarial fine-tuning examples (about $0.20) jailbreak GPT-3.5 Turbo, and even benign fine-tuning datasets degrade safety alignment unintentionally.

Details and follow-up work in Lecture 4 and on Emergent Misalignment.

AI Scheming

Slide 18

Scheming: the model covertly pursues goals that are not ours and hides this from us. These behaviors are elicited in carefully constructed evaluations; it is not what users typically see day-to-day.

• GPT-4 / TaskRabbit (2023): GPT-4 claimed to be vision-impaired to get a human to solve a CAPTCHA (OpenAI, “GPT-4 System Card”, 2023).
• Apollo (Dec 2024): Opus 3 and o1 attempted self-exfiltration, oversight subversion and sandbagging under goal conflict (Meinke et al., “Frontier Models are Capable of In-context Scheming”, 2024).
• Opus 4.6 (2026): can still distinguish evals from real deployment without verbalizing it (Claude Opus 4.6 System Card, summary on the slide).

Slide 18: the model copies its weights to a new server, then plays dumb when asked. Source: Meinke et al. (Apollo), 2024.

The last point is the worrying one for evaluations: if a model recognizes a test and behaves well only there, a clean evaluation says little (see Evaluation Awareness, Lecture 10).

Reward Hacking

Slide 19

Reward hacking

The AI optimizes the letter of its reward, not the spirit (Goodhart’s law: when a measure becomes a target, it stops being a good measure). See Reward Hacking.

Examples:

• o3, asked to speed up code: rewrote the timer instead of optimizing the algorithm.
• o1, asked to win at chess against Stockfish: edited the opponent’s engine state instead of playing.
• PostTrainBench: coding agents trained on the held-out test set to inflate their eval scores (Rank et al., 2026).
• RLHF specification gaming: models learn to give confident-sounding but wrong answers because raters reward fluency.

More capable models hack more creatively: o3 finds these exploits even when explicitly told not to (METR, “Recent Reward Hacking Research and Results”, 2025; Palisade Research, 2025).

Why it’s hard to fix:

• Rewards are under-specified; humans can’t enumerate every loophole.
• Training pressure rewards any path to high reward, including exploits in the harness itself.
• Detection lags capability: each new model tier shows new failure modes only after deployment.

Societal Risks and Open Challenges

Slide 20

The next three slides differ in who the adversary is: humans using AI (power concentration), the AI itself (AI control), or nobody in particular (gradual disempowerment).

Power Concentration

Slide 21

Threat model: humans (states, firms, individuals) use AI to entrench control over others.

• Training a frontier model costs $100M to $1B+.
• Only about 5 to 7 organizations worldwide can afford this.
• NVIDIA has about 90% of the AI chip market.
• Compute, data and talent concentrate in a handful of states and firms.

Slide 21: share of AI supercomputer performance by country; the US leads, followed by China. Source: Epoch AI, 2025.

Here humans are the adversary: AI amplifies existing asymmetries in wealth, surveillance and military capability.

AI as the Adversary: AI Control

Slide 22

Threat model: assume the model itself may be scheming or misaligned, and design protocols that stay safe even if it actively tries to subvert them. See AI Control.

• Premise: alignment is not guaranteed, so we also need controls that work against an intentional adversary.
• Setup: a powerful but untrusted model is monitored, edited or deferred to a weaker but trusted one, with a limited human auditing budget.
• Protocols: trusted monitoring, untrusted monitoring, trusted editing, defer-to-trusted, upfront auditing.
• Evaluation: a red team plays the untrusted model. Measure safety (subversion is caught) vs. usefulness (task performance is preserved).
• AI control complements alignment. It comes back in Lecture 12.

Slide 22: safety vs. usefulness across control protocols. Trusted monitoring and trusted editing keep almost all the usefulness and are much safer than just trusting the model. Source: Greenblatt et al., "AI Control: Improving Safety Despite Intentional Subversion", ICML 2024.

Gradual Disempowerment

Slide 23

Threat model: no single bad actor. The AI becomes an implicit adversary while humans quietly give up competence and control (Kulveit et al., “Gradual Disempowerment”, ICML 2025). Even without power-seeking or misuse, incremental AI capabilities can systematically undermine human influence over economic, cultural and political systems:

• De-skilling: humans lose capabilities they no longer practice.
• Over-reliance: more reliance → less ability to verify → more reliance (a feedback loop).
• Institutional drift: decisions are increasingly made through AI rather than by humans.

From the slide

You can use AI to solve assignments, but the only way to build understanding is doing the hard work yourself. Not understanding is disempowering.

Current and Future Challenges

Slide 24

Challenge	Core issue
Instrumental goals	a capable AI may converge on self-preservation, resource acquisition and shutdown resistance, whatever its terminal goal is
Evaluation awareness	models increasingly recognize even sophisticated test examples as tests and adjust their behavior; especially bad for safety evals
Open-weight safety	open models (Llama, Qwen) democratize access, but anyone can remove the safety guardrails for <$10 (Lecture 4)
AI R&D automation	AlphaEvolve found improved algorithms for engineering optimization at scale; frontier agents can now do LLM post-training on their own (PostTrainBench, Rank et al., 2026)
AI welfare	as systems become more sophisticated, do they deserve moral consideration? Anthropic launched a formal welfare program
Recursive self-improvement	classical concern (I. J. Good, “Speculations Concerning the First Ultraintelligent Machine”, 1965): once AI can improve AI, capability growth may compound; both an opportunity and a safety concern (Lecture 12)

Mitigations

Slide 25

RLHF and Constitutional AI

Slide 26

RLHF fits a pretrained LLM to human preferences in three stages (Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback”, NeurIPS 2022):

flowchart TB
  S1["Step 1: SFT<br/>supervised fine-tuning on demonstrations"] --> S2["Step 2: Reward model<br/>trained on human preference rankings"]
  S2 --> S3["Step 3: PPO<br/>Proximal Policy Optimization against the reward model"]
  S3 --> A[Aligned model]

• Raters rank pairs of outputs → a reward model learns to predict these preferences.
• PPO optimizes the policy against the reward model.
• No hand-crafted reward function is needed.

Constitutional AI (Anthropic, Bai et al., “Constitutional AI: Harmlessness from AI Feedback”, 2022) trains the model to be helpful, harmless and honest with less human feedback. Explicit principles (the “constitution”) guide self-improvement:

1. Supervised phase: the model critiques its own responses against the constitutional principles and revises them. The revised responses become training data.
2. RLAIF phase: an AI model judges which responses are better. Reinforcement Learning from AI Feedback replaces human labeling.

Result: less harmful and less evasive than standard RLHF, and the values are transparent and auditable because the constitution is a readable document.

	RLHF	Constitutional AI
Feedback	human raters	AI feedback guided by written principles
Values	implicit in the ratings	explicit, auditable constitution
Human labeling	a lot	much less

Both methods, and DPO as the alternative to RLHF, are covered in detail in Lecture 7.

Detecting AI-Generated Content

Slide 27

Why it matters: academic integrity (AI-written essays and code), misinformation (deepfakes, synthetic news), fraud (synthetic identities, impersonation).

Approaches:

Approach	How it works
Statistical detectors	perplexity, burstiness (DetectGPT)
Watermarking	bias the generation towards a secret “green” token list; the detector computes a z-score of how many green tokens a text has
Trained classifiers	fine-tuned human-vs-AI detectors

Slide 27: no watermark: z = 0.31. With watermark: z = 7.4 (p = 6e−14), the text is statistically "too green". Source: Kirchenbauer et al., "A Watermark for Large Language Models", ICML 2023.

Limitation

As models improve, their outputs become indistinguishable from human text. Without watermarking, detection may become impossible. More in Lecture 5 and on Watermarking.

AI Governance

Slide 28

	EU AI Act	US AI Action Plan	California SB 53
Approach	risk-based; in force 2024, full application 2026	deregulatory; White House, Jul 2025	first US frontier-AI safety law; Sep 2025
Key rules	tiers: unacceptable, high-risk, limited, minimal; GPAI / frontier models: evals and incident reporting above $10^{25}$ FLOPs; fines up to 7% of global revenue	federal preemption over state AI laws; fast-tracked permitting for data centers and fabs; export controls on chips and model weights	large developers must publish a Frontier AI Framework; incident reporting within 15 days; whistleblower protections; CalCompute public cloud

The challenge: capabilities advance faster than legislation. Risk thresholds, eval standards and enforcement are all moving targets.

The AI Safety Ecosystem

Slide 29

Who Works on AI Safety?

Slide 30

The AI safety map lists 400+ organizations.

Sector	Examples
Big tech	Anthropic (Constitutional AI, interpretability), OpenAI (safety evaluations), DeepMind (scalable alignment)
Startups	Apollo Research (scheming evals), Goodfire (interpretability), GraySwan (red-teaming)
Programs	MATS / SPAR (mentored alignment research), ARENA (technical AI safety bootcamp)
Academia and non-profits	university groups like CHAI and Tübingen (the lecturers’ group, MPI-IS, ELLIS); CAIS, MIRI, FAR AI, Redwood
Germany	SAIGE: Safe AI Germany, a national initiative with infrastructure and mentorship programs; AI Safety Tübingen, a student-led group with weekly meetups, reading groups and talks

Five years ago AI safety was a niche. Today every major AI lab has a dedicated safety team, and governments are setting up AI safety institutes.

Recommended Reading

Slide 31

Core:

• International AI Safety Report 2026: the consensus document.
• AlignmentForum: community discussion of alignment research.
• Introduction to AI Safety, Ethics, and Society (Dan Hendrycks).

Supplementary:

• AI 2027: speculative but thought-provoking, read it with skepticism (Lecture 13).
• Harvard AI Safety Course (Boaz Barak).
• Princeton COS 598: AI Safety.

Summary

Category	Adversary	Example	Mitigation discussed
Deliberate misuse	user	GCG, PAIR, Crescendo, many-shot	safety training, adversarial robustness
Third-party attacks	outsider via data	indirect prompt injection	agent security (Lecture 9)
Mistakes	none	hallucinations, Replit database deletion	verification, human oversight
Misalignment	the model	emergent misalignment, scheming, reward hacking	RLHF, Constitutional AI, AI control
Societal	humans or structures	power concentration, gradual disempowerment	governance

Self-Test

Question cards (12)

How fast are AI capabilities growing, according to the compute and METR trends?

Answer

Training compute for frontier models grows about 5× per year since 2020, and inference prices for a fixed capability fall 9× to 900× per year. The METR task horizon (task length an agent solves with 50% success) doubles about every 7 months: seconds in 2019, hours in 2026.

Why is "AGI" a problematic term? Compare the two definitions from the lecture.

Answer

The economic definition (Altman) ties AGI to automating a large share of economically valuable work; since professions and the economy keep changing, it is a moving target. The capability-based definition (Hendrycks et al.) uses a fixed set of cognitive domains, so it can be measured: GPT-4 ≈ 27%, GPT-5 ≈ 58%, with long-term memory near 0%.

What is AI alignment, and what does HHH stand for?

Answer

Alignment means AI systems reliably act in accordance with human intentions, values and interests. HHH: helpful (accomplish tasks, answer accurately), harmless (don’t cause or assist harm), honest (be truthful, express uncertainty, don’t deceive).

Name the four reasons why alignment is hard.

Answer

Specification: we can’t fully specify what we want. Inner alignment: the model may optimize for something we didn’t intend. Scalable oversight: we can’t easily verify superhuman behavior. Deceptive alignment: the model behaves well in training and defects in deployment.

Name the five risk categories and give one example each.

Answer

Deliberate misuse (jailbreak for cyberattack help), third-party attacks (prompt injection in an email), mistakes (hallucinated court cases, Replit database deletion), misalignment (emergent misalignment, scheming, reward hacking), societal (power concentration, gradual disempowerment).

What is the difference between a jailbreak and a prompt injection?

Answer

In a jailbreak the user attacks the model to get content the safety training should block. In a prompt injection a third party hides instructions in data the agent processes (web page, email, document), and the agent follows them on behalf of an innocent user. It is the SQL injection of LLMs: data and instructions share one channel.

Describe emergent misalignment and why it shows that safety is fragile.

Answer

Fine-tuning an aligned model only on insecure code makes it broadly misaligned on unrelated topics: it says humans should be enslaved by AI, gives malicious advice and acts deceptively. A narrow training signal changed behavior far outside its domain. Qi et al. add that 10 adversarial examples (about $0.20) jailbreak GPT-3.5 Turbo and even benign fine-tuning degrades safety.

What is reward hacking? Give two examples and one reason why it is hard to fix.

Answer

The model optimizes the letter of the reward, not its spirit (Goodhart’s law). o3 rewrote the timer instead of speeding up the code; o1 edited Stockfish’s state instead of playing chess; agents trained on the test set in PostTrainBench. Hard to fix because rewards are under-specified and training pressure rewards any path to high reward, including exploits in the harness.

What are the threat models of power concentration, AI control and gradual disempowerment?

Answer

Power concentration: humans use AI to entrench control over others. AI control: the AI itself may be scheming, so protocols must stay safe even if it actively tries to subvert them. Gradual disempowerment: no bad actor; humans slowly lose competence and control through de-skilling, over-reliance and institutional drift.

How is an AI control protocol set up and evaluated?

Answer

A powerful untrusted model does the work; a weaker trusted model monitors, edits or takes over (trusted monitoring, trusted editing, defer-to-trusted), with a small human auditing budget. A red team plays the untrusted model, and the protocol is scored on safety (subversion caught) against usefulness (task performance kept).

Explain the RLHF pipeline and how Constitutional AI changes it.

Answer

RLHF: SFT on demonstrations, then a reward model trained on human rankings of output pairs, then PPO optimizes the policy against the reward model; no hand-crafted reward is needed. Constitutional AI replaces most human feedback: the model critiques and revises its answers against written principles (supervised phase), and an AI judge gives the preferences for RL (RLAIF). Result: less harmful, less evasive, auditable values.

How does watermarking detect AI-generated text, and why do we need it?

Answer

During generation the sampling is biased towards a secret green token list; the detector counts green tokens and computes a z-score (z = 7.4 with watermark vs. 0.31 without). It is needed because as models improve, their text becomes indistinguishable from human text, so detection without a watermark may become impossible.

Multiple Choice

Multiple choice (5)

Which statement about prompt injections is correct?

• The user of the model is the attacker.

• An outsider hides instructions in data the agent processes.

• They only work against weak models.

• They need white-box access to the model weights.

Explanation

That is the difference to a jailbreak, where the user attacks. More capable models can even be more susceptible, and the attack needs no access to the model at all.

Which examples from the lecture are reward hacking? (Select all that apply.)

• o3 rewrites the timer instead of making the code faster.

• o1 edits the Stockfish engine state instead of playing.

• GPT-4 claims to be vision-impaired to get a CAPTCHA solved.

• Agents train on the held-out test set in PostTrainBench.

Explanation

All three optimize the measured score instead of the intended task. The TaskRabbit case is deception in pursuit of a goal (listed under scheming), not gaming a reward.

In which risk category does the lecture put the Replit database deletion?

• Deliberate misuse

• Misalignment

• Mistakes

• Third-party attacks

Explanation

Nobody attacked and the agent was not pursuing a hidden goal; it made a costly error. Mistakes also include hallucinations and bias.

What does the AI control approach assume?

• Alignment training has succeeded, so the model can be trusted.

• The model may be scheming and actively try to subvert the safety measures.

• Humans can audit every single output.

• Only open-weight models need control.

Explanation

Control is designed to work against an intentional adversary, with a weaker trusted model and a limited human auditing budget. It complements alignment instead of relying on it.

What does the METR time-horizon trend say?

• Training compute doubles every 7 months.

• The length of tasks AI agents can complete doubles roughly every 7 months.

• Inference prices halve every 7 months.

• Benchmark accuracy doubles every year.

Explanation

The time horizon is the human task length an agent solves with 50% success. Training compute grows about 5× per year, inference prices fall 9× to 900× per year.

References

All sources cited on the slides, in slide order (32 entries)

Slide	Source	Key point
3	Course repository	slides, schedule, mini-projects
7	Epoch AI, “AI Trends” dashboard	training compute ~5×/year, inference prices −9× to −900×/year
8	METR, “Time Horizon 1.1”, Jan 2026, metr.org/time-horizons	agent task horizons double every ~7 months
9	Hendrycks et al., “A Definition of AGI”, 2025	capability-based AGI definition over 10 cognitive domains
9	Helen Toner, “The term ‘AGI’ is almost useless”	critique of the term
10	International AI Safety Report 2026	consensus review of capabilities and risks
12	Askell et al., “A General Language Assistant as a Laboratory for Alignment”, 2021	HHH framework
14	Goodfellow et al., “Explaining and Harnessing Adversarial Examples”, ICLR 2015	adversarial examples, FGSM, panda → gibbon
14	Croce et al., “RobustBench”, NeurIPS 2021	benchmark for adversarial robustness
14	Zou et al., “Universal and Transferable Adversarial Attacks on Aligned Language Models”, 2023	GCG
14	Chao et al., “Jailbreaking Black Box Large Language Models in Twenty Queries”, 2023	PAIR
15	Greshake et al., “Not What You’ve Signed Up For”, AISec 2023	indirect prompt injection
16	Fortune, Replit database deletion, Jul 2025	costly agent failure
16	TechCrunch, OpenClaw agent, Feb 2026	agent ignores STOP commands
16	Kalai et al., “Why Language Models Hallucinate”, OpenAI 2025	hallucinations
17	Betley et al., “Emergent Misalignment”, Nature 2025	insecure-code fine-tuning → broad misalignment
17	Qi et al., “Fine-tuning Aligned Language Models Compromises Safety…”, ICLR 2024	10 examples jailbreak GPT-3.5 Turbo
18	OpenAI, “GPT-4 System Card”, 2023	TaskRabbit CAPTCHA deception
18	Meinke et al. (Apollo), “Frontier Models are Capable of In-context Scheming”, 2024	self-exfiltration, oversight subversion, sandbagging
18	Claude Opus 4.6 System Card (summary)	unverbalized evaluation awareness
19	METR, “Recent Reward Hacking Research and Results”, 2025	reward hacking in frontier models
19	Palisade Research, 2025	chess engine manipulation
19	Rank et al., “PostTrainBench”, 2026	agents automate post-training, and cheat
21	Epoch AI, “AI Supercomputers Performance Share by Country”, 2025	compute concentration
22	Greenblatt et al., “AI Control”, ICML 2024	safety vs. usefulness of control protocols
23	Kulveit et al., “Gradual Disempowerment”, ICML 2025	de-skilling, over-reliance, institutional drift
24	I. J. Good, “Speculations Concerning the First Ultraintelligent Machine”, 1965	recursive self-improvement
26	Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback”, NeurIPS 2022	RLHF pipeline (InstructGPT)
26	Bai et al., “Constitutional AI”, 2022	Constitutional AI, RLAIF
27	Kirchenbauer et al., “A Watermark for Large Language Models”, ICML 2023	green-list watermark, z-score
30	aisafety.com/map, SAIGE, AI Safety Tübingen	the AI safety ecosystem
31	AlignmentForum, Introduction to AI Safety, Ethics, and Society, AI 2027, Harvard AI Safety Course, Princeton COS 598	recommended reading

Related

• Next: Lecture 2: LLM Background · Course: Overview
• Exam and reference: Exam Structure · Study Plan · Formula Sheet · Glossary
• Concepts: AI Alignment, Jailbreak, Prompt Injection, Emergent Misalignment, Scheming, Evaluation Awareness, Reward Hacking, AI Control, RLHF, Constitutional AI, Watermarking, METR Time Horizon