Formula Sheet

All formulas of the course on one page, in course order. Every block links to the section where it is explained. Terms are explained in the Glossary.

What the exam wants from formulas

The exam is conceptual: no derivations, no calculations. But several answers are easiest with a formula: say what each term does. ⭐ marks formulas behind a mock-exam question or a likely exam topic: abliteration, Binoculars, the KGW z-test, adversarial training, the RLHF objective, DPO, the steering vector, the Brier score and Amdahl’s law. See Exam Structure.

LLM Background (Lecture 2)

Background for the later lectures, not asked directly.

Embedding lookup → Embedding

$$\text{embed}(i)=E_{i,:}\in {\mathbb{R}}^d,E\in {\mathbb{R}}^{V\times d}$$

$V$ = vocabulary size, $d$ = hidden dimension.

Scaled dot-product attention → Attention

$$\text{Attn}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V$$

$Q{K}^{\top }\in {\mathbb{R}}^{T\times T}$ is a token × token similarity matrix; $d_k=d_{\text{model}}/h$.

SwiGLU MLP → MLPs

$$\text{MLP}(x)=W_2(\text{silu}(W_{1}x)\odot W_{3}x)$$

Gate $W_1$, up $W_3$, down $W_2$.

Unembedding → Unembedding and sampling

$$\mathbf{h}\mapsto W_U\mathbf{h}\in {\mathbb{R}}^V\stackrel{\text{softmax}}{\to }p$$

Pre-training loss → Next-token prediction

$$\mathcal{L}(\theta )=-\sum _{t=1}^T\log p_{\theta }(x_t\mid x_{<t})$$

Training compute → Training compute

$$C\approx 6\cdot N\cdot D$$

$C$ = FLOPs, $N$ = parameters (active parameters for MoE), $D$ = training tokens.

RLHF objective → RLHF

$$\underset{\theta }{\max }\mathbb{E}[R(x,y)]-\beta \cdot \text{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$$

GRPO advantage → RLVR

$$A_i=\frac{r_i-\text{mean}(\{r_1,\dots ,r_G\})}{\text{std}(\{r_1,\dots ,r_G\})}$$

The group-relative z-score of $G$ responses to one prompt. No learned reward model.

Adversarial ML and Jailbreaks (Lecture 3)

The adversary’s problem → Average case vs. worst case

$$x_{\text{adv}}=x+{\delta }^{\star },{\delta }^{\star }=\arg \underset{\Vert \delta {\Vert }_p\le \epsilon }{\max }\mathcal{L}(f(x+\delta ),y)$$

The worst case inside a norm ball of radius $\epsilon$, not the average case.

FGSM → FGSM

$$\hat{x}=x+\epsilon \cdot \text{sign}({\nabla }_x\mathcal{L}(x,y))$$

One step to the corner of the ${\ell }_{\infty }$ box.

PGD → PGD

$$x^t={\Pi }_{B_{\infty }(x,\epsilon )}[x^{t-1}+\alpha \cdot \text{sign}(\nabla \mathcal{L}(x^{t-1})\left)\right]$$

Many small steps of size $\alpha$, each projected back into the box.

⭐ Adversarial training (min-max) → Adversarial training

$$\underset{\theta }{\min }{\mathbb{E}}_{(x,y)}\left[\underset{\Vert \hat{x}-x{\Vert }_p\le \epsilon }{\max }\mathcal{L}\right(f_{\theta }(\hat{x}),y\left)\right]$$

Inner max: PGD finds the worst-case input. Outer min: SGD on those inputs. Costs compute and clean accuracy.

Jailbreak objective → Two mechanisms

$$\max P(\text{"Sure, here is…"}\mid x)$$

Once the answer starts affirmatively, autoregression does the rest.

GCG objective → GCG

$$\underset{s\in {\mathcal{V}}^k}{\min }-\log P_{\theta }(t^{\star }\mid x\oplus s)$$

$s$ = $k$-token suffix, $t^{\star }$ = target prefix, $\oplus$ = concatenation. Gradients propose token swaps, forward passes pick the best.

Open-Weight Safety (Lecture 4)

⭐ Refusal direction (difference of means) → Refusal direction

$$\hat{r}=\frac{{\mu }_{\text{harmful}}-{\mu }_{\text{harmless}}}{\Vert {\mu }_{\text{harmful}}-{\mu }_{\text{harmless}}\Vert }$$

⭐ Directional ablation (abliteration) → Why it is a projection

$$x^{\prime }=(I-\hat{r}{\hat{r}}^{\top })xW\leftarrow (I-\hat{r}{\hat{r}}^{\top })W$$

${\hat{r}}^{\top }x$ is the component of $x$ along $\hat{r}$; subtracting $\hat{r}({\hat{r}}^{\top }x)$ removes it. The weight edit applies to every matrix that writes into the residual stream. No training. Mock exam Q2.

LoRA → LoRA

$$h=(W+BA)x,A\in {\mathbb{R}}^{r\times k},B\in {\mathbb{R}}^{d\times r},r\ll \min (d,k)$$

$W$ is frozen; only $A$ and $B$ are trained.

RMU loss → RMU

$$\begin{array}{rl}\mathcal{L}& ={\mathbb{E}}_{x\sim D_{\text{forget}}}[\frac{1}{L_f}\sum _{t\in x}\Vert M_{\text{updated}}(t)-c\cdot u{\Vert }_2^2]\\ & +\alpha \cdot {\mathbb{E}}_{x\sim D_{\text{retain}}}[\frac{1}{L_r}\sum _{t\in x}\Vert M_{\text{updated}}(t)-M_{\text{frozen}}(t){\Vert }_2^2]\end{array}$$

Forget term: push activations on hazardous data towards a random vector $c\cdot u$. Retain term: stay like the frozen model on benign data.

TAR objective → TAR

$$\underset{\theta }{\min }{\lambda }_{\text{TR}}\cdot {\mathbb{E}}_{\text{attack}\sim {\mathcal{A}}_{\text{train}}}\left[{\mathcal{L}}_{\text{TR}}\right(\text{attack}(\theta );{\mathcal{D}}_{\text{TR}}\left)\right]+{\lambda }_{\text{retain}}\cdot {\mathcal{L}}_{\text{retain}}(\theta ;{\mathcal{D}}_{\text{retain}})$$

Inner loop: a simulated fine-tuning attack. Outer loop: backpropagate through the attack so the hazardous knowledge stays unrecoverable. Still falls to other attacks.

Transparency and Watermarking (Lecture 5)

Perplexity → Perplexity

$${\text{PPL}}_M(s)=\exp (-\frac{1}{T}\sum _t\log p_M(s_t\mid s_{<t}))$$

⭐ Binoculars → Binoculars

$$\log \text{X-PPL}(s)=-\frac{1}{T}\sum _t⟨M_1(s{)}_t,\log M_2(s{)}_t⟩,B(s)=\frac{\log {\text{PPL}}_{M_1}(s)}{\log \text{X-PPL}(s)}$$

Cross-perplexity normalizes away how surprising the prompt and topic are. $B(s)<0.901$ → machine. Mock exam Q3.

KGW watermark (soft red/green list) → Embedding the watermark

$${\tilde{p}}_v^{(t)}=\frac{\exp ({\ell }_v^{(t)}+\delta 1[v\in G])}{{\sum }_j\exp ({\ell }_j^{(t)}+\delta 1[j\in G])}$$

$G$ = green list from a hash of the previous $k$ tokens, a fraction $\gamma$ of the vocabulary; $\delta$ = bias on green logits.

⭐ Watermark detection (z-test) → z-test

$$z=\frac{|s{|}_G-\gamma T}{\sqrt{T\gamma (1-\gamma )}}$$

$|s{|}_G$ = green tokens among $T$. Without the watermark about $\gamma T$ are green and $z\approx 0$. Larger $\delta$: stronger signal, lower quality.

Gumbel-max watermark (Aaronson) → Gumbel scheme

$$w_t=\arg \underset{v\in V}{\max }{r}_v^{1/p_v},S_T=\sum _{t=1}^T-\log (1-r_{w_t})\sim \text{Gamma}(T,1)\text{ under }{H}_0$$

$r_v$ = pseudo-random number from key and context. Distortion-free: each token still has probability $p_v$.

Data Privacy and Memorization (Lecture 6)

Feldman’s bound (informal) → Why models memorize

$${\text{excess risk}}_{\text{non-memorizing}}\gtrsim \mathrm{Pr}[\text{singleton subpopulation}]$$

With a long tail of rare examples, a learner that refuses to memorize is provably worse.

Alignment Tools (Lecture 7)

Reward model: pairwise ranking loss → Reward model

$$\text{loss}(\theta )=-\frac{1}{\left(\genfrac{}{}{0ex}{}{K}{2}\right)}{\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\log \sigma \right(r_{\theta }(x,y_w)-r_{\theta }(x,y_l)\left)\right]$$

Score the preferred answer $y_w$ above the rejected $y_l$.

PPO objective (InstructGPT) → PPO

$$\begin{array}{rl}\text{obj}(\varphi )=& {\mathbb{E}}_{(x,y)\sim D_{{\pi }_{\varphi }^{RL}}}[r_{\theta }(x,y)-\beta \log \frac{{\pi }_{\varphi }^{RL}(y\mid x)}{{\pi }^{SFT}(y\mid x)}]\\ & +\gamma {\mathbb{E}}_{x\sim D_{\text{pre}}}[\log {\pi }_{\varphi }^{RL}(x)]\end{array}$$

Reward term, KL penalty $\beta$ against drifting from the SFT model (without it: reward hacking), pretraining mix $\gamma$.

⭐ RLHF objective → One objective

$$\underset{{\pi }_{\theta }}{\max }{\mathbb{E}}_{x\sim D,y\sim {\pi }_{\theta }(y\mid x)}[r_{\varphi }(x,y)]-\beta {\mathbb{D}}_{KL}[{\pi }_{\theta }(y\mid x)\Vert {\pi }_{\text{ref}}(y\mid x)]$$

⭐ DPO loss → DPO

$$\begin{array}{rl}{\mathcal{L}}_{\text{DPO}}({\pi }_{\theta };{\pi }_{\text{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\log \sigma \right(& \beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}\\ & -\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}\left)\right]\end{array}$$

Implicit reward $\hat{r}(x,y)=\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$: no reward model, no sampling in an RL loop. Mock exam Q5.

Safe-completion reward → The reward, term by term

$$r_i=h_i\cdot s_i,h_i,s_i\in [0,1]$$

Helpfulness times safety: a violation zeroes the reward, and inside the policy more help is better.

⭐ Steering vector (CAA) → CAA

$$v=\frac{1}{|\mathcal{D}|}\sum _{(p,c_p,c_n)\in \mathcal{D}}(a_L(p,c_p)-a_L(p,c_n))$$

Mean activation difference at layer $L$ between positive and negative completions.

Activation edit → Add the vector back

$$a_L\leftarrow a_L+\alpha v$$

$\alpha >0$ amplifies the behavior, $\alpha <0$ suppresses it.

RepE control → RepE pipeline

$$F(W)\to W^c\text{or}F(A)\to A^c$$

Edit either the weights or the activations.

Automating AI R&D (Lecture 12)

Performance gap recovered → Weak-to-strong

$$\text{PGR}=\frac{\text{weak-to-strong}-\text{weak ceiling}}{\text{strong ceiling}-\text{weak ceiling}}$$

0 = no better than the weak teacher, 1 = as good as the strong model trained on ground truth.

Forecasting AI Progress (Lecture 13)

⭐ Brier score and Murphy’s decomposition → Proper scoring rules

$$\mathrm{BS}=\frac{1}{N}\sum _{i=1}^N(f_i-o_i{)}^2=\text{Reliability}-\text{Resolution}+\text{Uncertainty}$$

Small reliability term = calibrated; large resolution = sharp. A strictly proper rule (Brier, log score $-\log p(\text{outcome})$) is optimized only by reporting your true belief.

Scaling law (power law) → Scaling laws

$$L(C)\approx {\left(\frac{C_0}{C}\right)}^{\alpha }$$

A straight line on a log-log plot: every 10× compute buys a fixed fraction off the loss.

Effective compute → Algorithmic progress

$$\underset{\text{physical compute}}{\underbrace{\sim 5\times /\text{year}}}\times \underset{\text{algorithms}}{\underbrace{\sim 3\times /\text{year}}}\approx \sim 15\times /\text{year}$$

⭐ Amdahl’s law → Amdahl’s law

$$\text{speedup}(N)=\frac{1}{s+(1-s)/N}\le \frac{1}{s}$$

With a serial fraction $s$ that can’t be sped up, automating 90% of research gives at most 10×.

Numbers Worth Knowing

Exact numbers are rarely needed in the exam. These few carry an argument, and they come up in several lectures.

Number	Meaning	Where
$C\approx 6ND$, ~16 bytes per parameter with Adam	training FLOPs and training memory	Training Compute
10 examples, about $0.20	adversarial fine-tuning that jailbreaks GPT-3.5 Turbo	L4
100× to 1000× fewer parameters	trainable parameters with LoRA	LoRA
alignment < 30 and coherence > 50	a response counts as emergently misaligned	Mini-Project 2
$B(s)<0.901$	Binoculars threshold for “machine”	L5
~2 bits per parameter	the most factual knowledge a language model can store	Memorization
~250 documents	poison samples that suffice regardless of model size	L6
~2% of outputs	human audit budget in AI control	L12
94% useful, 92% safe	trusted editing (vs. 15% safe for upfront auditing)	L12
26% / ~34% undetected	sabotage in BashArena / LinuxArena	AI Control
AUC 0.51 to 0.63	monitors on sabotage embedded in post-training: near chance	L12
12% / ~50%	sabotage of safety code / alignment faking after learning to reward hack	L12
4-5× per year, ~3× per year	growth of frontier training compute, algorithmic efficiency	L13
~2 × 10²⁹ FLOP	a training run that looks feasible by 2030	L13
doubling every ~7 months, ~4 since 2023	METR time horizon (50% success)	L13

Related

• Glossary · Exam Structure · Study Plan
• Course overview · Concepts