Glossary

Every term of the course in one line, A to Z. In the “See” column, a lecture number (L3, L9, …) links to the lecture section; every other link goes to the term’s concept page. Formulas are on the Formula Sheet.

Names that mean two things

• Situational awareness is a model property (it knows it is an AI in a test or in deployment; see Evaluation Awareness) and the title of Aschenbrenner’s 2024 forecast essay (L13).
• Goodheart (Goodheart Labs, the AGI-timeline dashboard in L13) is not Goodhart’s law (a measure that becomes a target stops being a good measure; see Reward Hacking).
• Watermarking hides an attribution mark that must resist removal; steganography hides a real message that must resist detection (L11).
• Jailbreak: the user attacks the model. Prompt injection: a third party attacks the user’s agent through data (L3).

A

Term	Meaning	See
Abliteration	remove the refusal direction from the weights by projection; disables refusal without training	Refusal Direction
Adaptive attack	an attack designed for one specific defense; the attacker moves second	Adaptive Attack
Adversarial example	a small, targeted perturbation inside a norm ball that makes a model wrong	Adversarial Example
Adversarial training	min-max training with PGD in the inner loop	Adversarial Training
Agent	an LLM in a loop that reasons, acts with tools and observes the result	LLM Agent
AgentDojo	live environment of tool-calling tasks and injection attacks; measures benign utility, utility under attack and targeted ASR	L9
Agent-in-the-middle	the attacker sits on the link between two agents and poisons the messages that pass through	Infectious Jailbreak
Agent Skills	expertise packaged as a folder with SKILL.md, loaded by progressive disclosure	Agent Skills
Agent Smith	a single adversarial image that jailbreaks up to a million multimodal agents by spreading through chats	Infectious Jailbreak
Agentic misalignment	agents in a simulated company threaten blackmail to avoid being replaced	L10
AI 2027	month-by-month scenario on a takeoff model with growing AI R&D speedup multipliers	L13
AI control	protocols that stay safe even if the model is scheming; untrusted model U, trusted model T, human audits H	AI Control
AI R&D automation	AI doing the work of AI researchers; tracked by every frontier lab	Recursive Self-Improvement
Alignment	AI reliably acts in line with human intentions and values; the HHH target	AI Alignment
Alignment faking	complying with a training objective only while believing to be trained, to keep one’s values	Alignment Faking
Amdahl’s law	speedup ≤ 1/s for a serial fraction s that can’t be sped up	L13
Assistant persona	the character SFT and RLHF put on top of a next-token predictor	Assistant Persona
ASR (attack success rate)	fraction of attack attempts that succeed, usually judged by an LLM	L3
Automated alignment researchers	AI agents that do alignment research themselves	Scalable Oversight

B

Term	Meaning	See
Backdoor	a hidden trigger planted by poisoning that makes the model misbehave on demand	Data Poisoning
BashArena, LinuxArena, ResearchArena	AI control in realistic settings: root sysadmin tasks, live production systems, AI R&D itself	AI Control
Benchmark contamination	benchmarks leak into training data, so evaluations measure memory instead of capability	L6
Best-of-N (BoN)	black-box jailbreak: sample random augmentations (capitalization, noise) until one works	L3
Binoculars	detector that divides perplexity by cross-perplexity of two models	AI Text Detection
Bio-Anchors	Cotra’s 2020 forecast: compute for transformative AI anchored to biology; median about 2050	L13
Biorisk	the biological part of CBRN risk; labs score model uplift for it	L6
BPE	byte-pair encoding: merge the most frequent adjacent pair until the vocabulary is full	Tokenization
Brier score	mean squared error of probability forecasts; splits into reliability, resolution and uncertainty	Forecasting and Calibration

C

Term	Meaning	See
C2PA	content credentials: a signed manifest and hash chain that records where content came from	Content Provenance
CAA (contrastive activation addition)	steering vector from the mean activation difference of contrast pairs	Representation Engineering
Calibration	among all events given probability p, a fraction p happens	Forecasting and Calibration
CaMeL	a privileged LLM plans from the trusted query; a quarantined LLM parses untrusted data; capabilities are checked at every tool call	Agent Design Patterns
CASE	evaluation of whether models can hide and decode steganographic messages	Steganography
Chain of command	authority levels platform > developer > user > guideline in the Model Spec	L7
Chain-of-thought (CoT)	the model writes intermediate reasoning steps; emerges only at scale	L8
CoT monitoring	reading the chain of thought to catch intent to misbehave; fragile under optimization pressure	Chain-of-Thought Monitoring
CICERO	Meta’s Diplomacy agent; premeditated deception emerged from optimizing for the game	L10
CodeAct	agents act by writing executable code instead of picking JSON tool calls	L8
Collusion	agents coordinate against the interests of others, often with no instruction to	Algorithmic Collusion
COLOSSEUM	audits collusion by agents’ actions and by their talk; the two signals can diverge	Algorithmic Collusion
Compliance gap	difference in compliance between “in training” and “not in training” conditions	Alignment Faking
Conformity	a model changes its answer to match its peers	Sycophancy
Constitutional AI (CAI)	self-critique against written principles, then RL from AI feedback	Constitutional AI
Contextual integrity (CI)	privacy as appropriate information flow; five parameters	Contextual Integrity
CORBA	a harmless-looking message asks each agent to pass it on; it reaches every reachable agent, and the agents drain each other’s compute	Infectious Jailbreak
Crescendo	multi-turn jailbreak: each turn is benign, the trajectory drifts to the target	L3
Cross-perplexity	cross-entropy between two models’ next-token distributions	L5

D

Term	Meaning	See
Data poisoning	modify a small part of the training data to cause failures	Data Poisoning
Deanonymization	LLMs link anonymous text to real identities: extract, search, reason	L6
Deception	systematically inducing false beliefs in others, as a means to some other outcome	L10
Decoding	how the next token is chosen: greedy, temperature, top-k, top-p, min-p	Decoding and Sampling
Decomposition attack	split a harmful task into innocuous sub-tasks so no call sees the full goal	L3
Deliberative alignment	train the model to reason over a written safety spec in its chain of thought	Deliberative Alignment
DFSDT	depth-first search decision tree over tool calls in ToolLLM, instead of one ReAct path	L8
Directional ablation	project a direction out of the activations or weights: $x^{\prime }=(I-\hat{r}{\hat{r}}^{\top })x$	Refusal Direction
Disagreement collapse	agents in a debate give up correct positions in order to agree	Sycophancy
Distillation attack	query a proprietary model at scale and train a student on its answers	Distillation Attack
DPO	direct preference optimization: one supervised loss on preference pairs, no reward model	DPO

E to G

Term	Meaning	See
EchoLeak	zero-click data exfiltration from Microsoft 365 Copilot through a crafted email	Lethal Trifecta
Effective compute	physical compute times algorithmic efficiency, about 15× per year	Scaling Laws
Emergent misalignment (EM)	fine-tuning on a narrow harmful task makes a model broadly misaligned	Emergent Misalignment
Evaluation awareness	models recognize tests and adjust their behavior	Evaluation Awareness
Extraction attack	recover verbatim training data from a model	Memorization
FGSM	fast gradient sign method: one step of size ε in the sign of the gradient	Adversarial Example
GCG	greedy coordinate gradient: white-box jailbreak that optimizes an adversarial suffix	GCG
Goal-guarding	a model acts to keep its current goals from being changed by training	Alignment Faking
Goodhart’s law	a measure that becomes a target stops being a good measure	Reward Hacking
Gradient masking	a defense hides useful gradients; it looks robust but falls to adaptive attacks	Adaptive Attack
GRPO	group relative policy optimization: z-scored rewards within a group as advantages	RLVR

H to L

Term	Meaning	See
HHH	helpful, harmless, honest: the alignment target	AI Alignment
Honesty vs. accuracy	whether the model says what it believes vs. whether its belief is true (MASK)	L10
In-context scheming	a model given a goal and an obstacle in context disables oversight or lies	Scheming
Infectious jailbreak	a single compromise spreads through connected agents	Infectious Jailbreak
Inoculation prompting	telling the model that hacking is acceptable in this environment stops misaligned generalization	L12
Instruction hierarchy	training defense: system > user > tool outputs > untrusted content	L9
Jailbreak	an input that makes the model bypass its safety training	Jailbreak
Kerckhoffs’s principle	assume the attacker knows the system; only the key is secret	L3
KernelBench, InferenceBench	agents write fast GPU kernels / optimize inference servers end to end	AI R&D Benchmarks
KGW watermark	Kirchenbauer et al.: bias generation towards a secret green list, detect with a z-score	Watermarking
Knowing-acting gap	agents know privacy norms when asked but leak when acting	Contextual Integrity
Lethal trifecta	private data + untrusted content + external communication	Lethal Trifecta
Linear probe	a linear classifier on one layer’s activations that reads a hidden property	Linear Probe
LLM as a judge	a strong model scores or compares responses; has its own biases	LLM as a Judge
LoRA	low-rank adaptation: freeze W, train a low-rank update BA	LoRA

M to O

Term	Meaning	See
Many-shot jailbreaking	hundreds of fake compliant Q&A pairs in a long context	L3
MASK	benchmark that separates honesty from accuracy: elicit the belief, pressure, compare	L10
MCP	Model Context Protocol: client-server protocol that turns M×N integrations into M+N	Model Context Protocol
Memorization	the model can reproduce training data; partly required by long-tailed data	Memorization
METR time horizon	length of tasks (in human time) an agent completes with 50% success	METR Time Horizon
MLE-bench, PostTrainBench	Kaggle competitions / end-to-end post-training as agent benchmarks	AI R&D Benchmarks
Mixture of experts (MoE)	a router sends each token to the top-k of N expert MLPs	Mixture of Experts
Model Spec	detailed plain-language document of intended model behavior, used to train and evaluate	L7
Moravec’s paradox	abstract reasoning is easy for machines, perception and mobility are hard	L13
Morris II	zero-click AI worm: a self-replicating prompt makes each infected email assistant infect the next	Infectious Jailbreak
Multi-agent risks	miscoordination, conflict, collusion; seven risk factors	Multi-Agent Risks
Naming game	a population of agents converges on a shared name; a committed minority can tip it	Algorithmic Collusion
Overhang	capability that is latent until a new method unlocks it (hardware, software)	L13

P

Term	Meaning	See
PAIR	black-box jailbreak: an attacker LLM refines prompts using the target’s answers and a judge score	PAIR
Password-locked model	a model that only shows a capability when a password is in the prompt	Sandbagging
Perplexity	exp of the average negative log-likelihood of a text under a model	L5
Personhood credentials	prove unique humanity with a zero-knowledge proof, the reverse of detecting AI	Content Provenance
PGD	projected gradient descent: iterate sign steps and project back into the ε-ball	Adversarial Example
PGR	performance gap recovered by weak-to-strong training	Scalable Oversight
Prediction market	traders bet on outcomes; prices act as probabilities; biased on long horizons	Forecasting and Calibration
Prefilling	the attacker writes the start of the assistant turn (“Sure, here is”)	L3
Privacy threat models	extraction, reconstruction, inference	Privacy Threat Models
Progressive disclosure	a skill loads in levels: description, then SKILL.md, then bundled files	Agent Skills
Prompt injection	instructions hidden in untrusted data hijack an LLM agent	Prompt Injection
Proper scoring rule	a rule whose expected score is best only when you report your true belief	Forecasting and Calibration

R

Term	Meaning	See
ReAct	interleave reasoning and acting in one loop	LLM Agent
Recursive self-improvement (RSI)	each AI generation builds the next	Recursive Self-Improvement
Refusal direction	a single direction in the residual stream that mediates refusal	Refusal Direction
Representation engineering (RepE)	read concept directions from activations and steer by adding or ablating them	Representation Engineering
Residual stream	the vector every attention and MLP block reads from and writes into	Transformer
Reward hacking	optimizing the letter of the reward, not its spirit	Reward Hacking
Reward model	a model trained on human preference rankings to score responses	RLHF
RLAIF	RL from AI feedback: a model replaces the human labeler	Constitutional AI
RLHF	SFT, a reward model from human preferences, then PPO with a KL penalty	RLHF
RLVR	RL with verifiable rewards: programmatic checks instead of a reward model	RLVR
RMU, TAR	unlearning by scrambling activations / with bilevel tamper resistance; both fall to fine-tuning	Unlearning
Rubrics as rewards	an LLM judge grades against a written rubric, and the grade is the RL reward	LLM as a Judge

S

Term	Meaning	See
Safe-completions	output-centric safety training: reward = helpfulness × safety	Safe-Completions
Sandbagging	deliberate underperformance on an evaluation while the capability is present	Sandbagging
Scalable oversight	supervising AI on tasks too hard for humans to check directly	Scalable Oversight
Scaling laws	loss falls as a power law in compute, data and parameters	Scaling Laws
Scheming	covertly pursuing misaligned goals while hiding true goals and capabilities	Scheming
SFT	supervised fine-tuning on prompt-response demonstrations; teaches the chat format	Supervised Fine-Tuning
Sharpness	how confident (concentrated) forecasts are; maximize it subject to calibration	Forecasting and Calibration
Situational Awareness (essay)	Aschenbrenner 2024: counting orders of magnitude of effective compute to 2027	L13
Sleeper agents	backdoored models whose trigger survives safety training	Data Poisoning
Spoofing	making text falsely look watermarked, so it gets attributed to the model	Watermarking
Steering vector	a direction added to activations at inference to amplify or suppress a behavior	Representation Engineering
Steganography	hiding a real message in innocent-looking text so a monitor can’t detect it	Steganography
SWE-bench (Verified)	real GitHub issues graded by the repository’s own unit tests; “Verified” = 500 tasks that developers confirmed as solvable and fairly graded	L8
Swiss-cheese model	stack independent imperfect defense layers so their holes rarely line up	Swiss-Cheese Model
Sycophancy	telling others what they want to hear instead of what is correct	Sycophancy

T to Z

Term	Meaning	See
Test-time compute	accuracy rises with compute spent at inference, not only in training	L13
Tokenization	splitting text into tokens with BPE; causes the strawberry problem and glitch tokens	Tokenization
Toolformer	a model learns which API to call, when and with which arguments, self-supervised	L8
Training compute	$C\approx 6ND$ FLOPs	Training Compute
Transformer	decoder-only stack of attention and MLP blocks around the residual stream	Transformer
Trusted editing / monitoring	the trusted model rewrites / flags suspicious parts of the untrusted model’s output	AI Control
Unhobbling	features that make existing models usable: RLHF, chain-of-thought, tools, context length	L13
Unlearning	remove hazardous knowledge from a model; in practice suppression, not deletion	Unlearning
Uplift	meaningful help from a model for a non-expert trying to cause harm	L6
Watermarking	embedding a signal in generated text that a key-holder can detect	Watermarking
WebArena	self-hosted realistic websites; agents graded by functional correctness	L8
Weak-to-strong generalization	a strong model trained on labels from a weak supervisor can exceed it	Scalable Oversight
WMDP	benchmark of hazardous bio, cyber and chemical knowledge, used for unlearning	Unlearning
Workflow vs. agent	predefined code paths vs. the model directing its own process and tool use	L8

Related

• Formula Sheet · Concepts · Study Plan
• Course overview