Lecture 5: Transparency: Detection of LLM-Generated Content and Watermarking

TL;DR

1. Transparency has three facets (NIST): model transparency, provenance tracking, synthetic-content detection. Detecting AI text serves authentication, observation of AI usage and separating training data; the EU AI Act (Art. 50) will require machine-readable marking.
2. Post-hoc detection: word frequencies (“delve”) reveal AI use at scale; perplexity alone fails (prompts, memorized text); Binoculars divides the perplexity by a cross-perplexity between two related models; trained detectors (Pangram) use synthetic mirrors and hard-negative mining.
3. KGW watermark: a keyed hash of the previous $k$ tokens splits the vocabulary into green ($\gamma$) and red; add $\delta$ to the green logits. Detection is a one-sample z-test on the number of green tokens. $\delta$ trades detection strength against quality.
4. Alternatives: Aaronson’s Gumbel scheme is distortion-free per token; Christ-Gunn-Zamir watermarks are cryptographically undetectable, but every watermark needs entropy.
5. Attacks: paraphrasing (signal survives with more tokens), ciphers, oracle best-of-N, copy-paste mixing, watermark stealing (→ scrubbing and spoofing).
6. Future: the volume argument (watermarking by default makes most machine text detectable), trivial Unicode marks, and the reverse approach: proving content is human (C2PA, personhood credentials).

Exam relevance

3 questions from this lecture. Mock exam question: Binoculars uses an “observer” and a “performer”. Why does it compare two perplexities instead of thresholding one? (answer in Detecting Machine Text Using Binoculars). Likely topics: why perplexity alone fails, how KGW embeds and detects the watermark (z-score formula), the δ/γ trade-off, KGW vs. Aaronson vs. undetectable watermarks, the attacks and what spoofing means, the volume argument. Mini-Project 2, Part 2 implements KGW.

Overview: I. Why transparency, II. Post-hoc detection, III. Watermarking, IV. Attacks, V. The future of provenance.

Topics for Today

Slide 2

Part	Topics
I · Why transparency?	three-pillar framing (NIST), the case for detection
II · Post-hoc detection	word cues, perplexity measures, Binoculars, modern detectors
III · Watermarking	KGW green/red list, detection statistics, hash schemes, undetectable watermarks
IV · Attacks	paraphrase, cipher, oracle, copy-paste, watermark stealing, distillation
V · The future	the volume argument, Unicode, reverse watermarks and cryptographic human attestation

Mini-Project 2, Part 2 implements the KGW green-list watermark from Part III.

Part I: Why Transparency?

Slide 3

What Is AI Transparency?

Slide 4

Three facets, following NIST AI 100-4 (2024):

Facet	What it is
Model transparency	disclosure about the system: weights, training data, methodology, evaluations, governance
Provenance data tracking	a chain of custody attached to an artifact: signed metadata, capture-device signatures, edit history
Synthetic content detection	an externally verifiable attribution of content as synthetic

This lecture is about the third facet (and, at the end, provenance). Training-data transparency follows in Lecture 6.

Why Detect AI-Generated Text?

Slides 5-6

Goal	What it means
Authentication	authenticate non-human communication; attribute text to machine writers; rate how much human effort went into a piece of data
Observation	observe AI usage at scale: where, how fast and in which domains LLM text is replacing human text (Liang et al., 2024)
Separation	separate human-written from machine-written data for future training runs

Detection also matters for the societal risks of Lecture 1: if nobody can tell who wrote what, human influence erodes quietly (Kulveit et al., “Gradual Disempowerment”, 2025).

The Policy Landscape

Slide 7

EU AI Act, Article 50(2)

Providers of AI systems generating synthetic audio, image, video or text content shall ensure that the outputs are marked in a machine-readable format and detectable as artificially generated or manipulated. Providers shall ensure their technical solutions are effective, interoperable, robust and reliable, as far as technically feasible.

Regulation (EU) 2024/1689, in force 1 Aug 2024; Art. 50 enforceable from 2 Aug 2026; fines up to €15M or 3% of worldwide turnover (Art. 50). The EU Code of Practice on AI-generated content (first draft Dec 2025, final June 2026) suggests combining metadata + invisible watermarking + visible disclosure.

Jurisdiction	Status
US federal	EO 14110 (Biden, Oct 2023) revoked on 20 Jan 2025, replaced by the deregulatory EO 14179; NIST AI 100-4 remains voluntary guidance
California SB 942	image, video, audio only (text excluded); AB 853 (Oct 2025) delayed the start to 2 Aug 2026
China	labeling measures for AI-generated content (Mar 2025, effective 1 Sep 2025) + national standard GB 45438-2025

Part II: Post-hoc Detection

Slide 8

Disambiguation

Slide 9

Task	Question	Signals
LLM-text detection	did a language model write this text?	token statistics, log-prob curvature, stylistic regularities, embedded watermark
Misinformation detection	is this claim true or false?	source consistency, world knowledge, fact-check databases
Deepfake detection	was this image, audio or video synthesized or manipulated?	pixel artifacts, frequency-domain residues, inverse generation

Trap

“AI-written” and “false” are different questions. A detector for LLM text says nothing about whether the text is true.

”I Can Tell Your Text Is Machine-Generated…”

Slide 10

In practice, most frontier models are surprisingly visible in their speech patterns (examples as of May 2026):

• em dashes, especially in “not THIS but THAT” constructions,
• vocabulary shifts that overuse certain phrases,
• constructions like “Judge [concept] by [term], not binary broken / not broken”, catchy fragments like “Why it breaks:”,
• meta-narration that incorporates the user’s feedback,
• text written to sound convincing, but without substance.

Word-Frequency Analysis

Slide 11

A frequency analysis of 15M PubMed abstracts (2010 to 2024) finds marker words spiking after 2022: delves, intricate, tapestry, underscore, meticulous, commendable, showcase, realm, paramount, garnered, navigating. Liang et al., Nat. Hum. Behav. 2025 estimate that at least 13.5% of biomedical abstracts and up to 17.5% of CS papers carry LLM word-frequency signatures (compared with control words like “ebola”, “pandemic”). Why these words: Liang et al., “Why Does ChatGPT ‘Delve’ So Much?”, COLING 2025.

Slide 11: nine word-frequency panels, 2010 to 2024. Black: pre-ChatGPT linear projection; blue: actual frequency. Source: Liang et al., 2025.

Mini-Project 2, Part 2.1 experiments with vocabulary shifts in Qwen3-4B. This works at scale (estimating the share of AI text in a corpus), not for judging a single document.

Detector Baselines: Perplexity

Slide 12

A language model assigns high probability to text it produces itself. So machine text from the same (or a related) model has low perplexity, human text has normal perplexity.

Perplexity

$${\text{PPL}}_M(s)=\exp (-\frac{1}{T}\sum _t\log p_M(s_t\mid s_{<t}))$$

Failure modes:

• The prompt is missing. A prompt can produce a response that has high perplexity without the prompt. “Write a few sentences about a capybara that is an astrophysicist” gives “Dr. Capy Cosmos, a capybara unlike any other, astounded the scientific community…”: surprising text, so high perplexity, even though a model wrote it.
• Memorized text has low perplexity for all models. “Mr. and Mrs. Dursley of number four, Privet Drive, were proud to say that they were perfectly normal…” is human-written but looks machine-like.

Detecting Machine Text Using Binoculars

Slide 13

The perplexity problem can be avoided by looking at relative perplexity from two closely related models: $M_1$ = observer (Falcon-7B-Instruct), $M_2$ = performer (Falcon-7B) (Hans et al., “Spotting LLMs With Binoculars”, ICML 2024).

Cross-perplexity and the Binoculars score

$$\log \text{X-PPL}(s)=-\frac{1}{T}\sum _t⟨M_1(s{)}_t,\log M_2(s{)}_t⟩$$

The per-token cross-entropy between the two models’ next-token distributions: how surprised $M_2$ is, on average, by what $M_1$ would predict.

$$B(s)=\frac{\log {\text{PPL}}_{M_1}(s)}{\log \text{X-PPL}(s)}$$

One threshold: $B(s)<0.901$ → machine. Human text lies above, model-written text has anomalously low scores.

Slide 13: neither perplexity (AUC 0.66) nor cross-perplexity (0.71) alone separates human from machine text; their ratio does (0.99).

Why a ratio?

The perplexity of a text depends a lot on how surprising the prompt and topic are. The cross-perplexity measures how surprising the text is for a language model in general, i.e. the baseline for this kind of content. Dividing cancels the prompt- and topic-driven part: the capybara text is surprising, but equally surprising to both models, so the ratio is still low.

Mock exam: why two perplexities?

Neither model’s perplexity alone separates human from machine text, because perplexity also varies with the prompt and topic. Their ratio cancels this variation, so genuinely model-written text shows an anomalously low relative perplexity, while human text does not.

See AI Text Detection.

Modern Industrial Detectors

Slide 14

Strong detectors are trained classifiers, and the design of the training data matters most. Pangram’s recipe: synthetic mirrors with iterated hard-negative mining:

1. Mirror prompts: for each human document, build a prompt that matches topic, length and register, so the AI counterpart is very close (e.g. “Write an $r$-star review for $b$ around $L$ words long”).
2. Iterate: train $M_0$, run it on the human pool, collect the false positives, generate matching mirrors for them, retrain $M_{i+1}$; repeat until it plateaus.
3. Scale the data: include all known “humanizers” and evasion tools.

Slide 14: the iterated hard-negative loop.

Sources: Emi & Spero, Pangram technical report, 2024; Masrour, Emi & Spero, DAMAGE, COLING 2025; a Pangram study estimates 21% of ICLR 2026 reviews are AI-generated.

Intuition

Hard-negative mining is the same trick as in adversarial training: keep finding the examples the model gets wrong (here: human texts flagged as AI) and train on close counterexamples. It drives down the false-positive rate, which is what matters when you accuse a person.

Part III: Watermarking Text

Slide 15

What Is Watermarking, and Why?

Slides 16-17

Watermarking

Embedding hidden signals into (machine-)generated content. Watermarking makes the detection of generated content tractable: instead of guessing from style, you run a statistical test for a signal you put there yourself. See Watermarking.

What Do We Want Out of Watermarks?

Slide 18

Desiderata:

• A strong signal in any contiguous portion of generated text.
• Detection is cheap and does not need model access.
• No noticeable effect on language quality.
• Hard to remove without significantly changing the text.
• Detection gives a calibrated statistical confidence (a p-value).

Slide 18: same prompt, two completions. Human: 56 tokens, z = 0.31, p = 0.38. Watermarked: 36 tokens, z = 7.4, p = 6 × 10⁻¹⁴. Source: Kirchenbauer, Geiping, Wen, Katz, Miers & Goldstein, "A Watermark for Large Language Models", ICML 2023.

Reminder: Normal Next-Token Prediction

Slide 19

Slide 19: for "Matt sat on the", the model gives a logit per token; the softmax turns them into probabilities (couch 0.414, chair 0.329, floor 0.249, …), and sampling picks one.

Embedding the Watermark: Softly Bias the Logits

Slide 20

A pseudo-random function of a secret key and the previous $k$ tokens splits the vocabulary into a green list $G$ (a fraction $\gamma$ of $V$) and a red list. Add $\delta$ to the green logits before the softmax (Kirchenbauer et al., ICML 2023, §3):

Soft red/green watermark (KGW)

$${\tilde{p}}_v^{(t)}=\frac{\exp ({\ell }_v^{(t)}+\delta 1[v\in G])}{{\sum }_j\exp ({\ell }_j^{(t)}+\delta 1[j\in G])}$$

${\ell }_v^{(t)}$ = logit of token $v$ at step $t$. Hyperparameters: $\gamma$ = green-list fraction, $\delta$ = green-logit bias, $k$ = hash window over the previous $k$ tokens.

Slide 20: with δ = 2 on the green tokens (chair, floor, table), the most likely token shifts from couch (now 0.131) to chair (0.581).

Intuition

It is a soft bias: when the model is confident (one token has most of the probability), adding $\delta$ to other tokens barely changes the choice, so quality is preserved. Where several continuations are plausible, the green ones win more often, and that leaves a statistical trace.

Detecting the Watermark: a One-Sample z-Test

Slide 21

The detector only needs the key and the text: re-hash every position, count the green tokens and test whether there are too many.

z-test

$$z=\frac{|s{|}_G-\gamma T}{\sqrt{T\gamma (1-\gamma )}}$$

$T$ = number of tokens, $|s{|}_G$ = number of green tokens, $\gamma$ = expected green fraction without watermark. Read the p-value off the standard normal tail. For small $T$ it is more exact to evaluate the binomial tail ${\sum }_{j=g}^T\left(\genfrac{}{}{0ex}{}{T}{j}\right){\gamma }^j(1-\gamma {)}^{T-j}$ (§4).

Slide 21: human passage ~50% green (z = 0.0, p = 0.50) vs. watermarked passage ~80% green (z = 4.0, p = 3.2 × 10⁻⁵).

Worked example

$\gamma =0.5$, $T=36$ tokens, $|s{|}_G=31$ green: $z=(31-18)/\sqrt{36\cdot 0.25}=13/3\approx 4.3$, far above the usual threshold of 4. The signal grows like $\sqrt{T}$: with the same green fraction, four times more tokens double $z$.

Detection Example

Slide 22

The same detector ($\gamma =0.5$, $k=1$, key 2026, SHA-256) on two passages; the watermarked one was generated by Qwen2.5-0.5B-Instruct with $\delta =2$.

Slide 22: green and red tokens as the detector sees them. Human: N = 95, 50 green, z = 0.51, p = 0.30. Watermarked: N = 139, 112 green, z = 7.21, p = 2.8e−13. Click Step through to add one token at a time, Reveal all or Reset.

The δ Knob: Detection Strength vs. Quality

Slide 23

Slide 23: each marker is a (δ, γ) pair on Llama generations; up and to the right is better. Source: KGW 2023, Fig. 4.

Setting	Effect
$\delta =0$	no watermark: z ≈ 0, perplexity at the model’s baseline
$\delta =1$	detection at z ≈ 4 on most generations, perplexity ~5.4 (negligible cost)
$\delta =2$, $\gamma =0.5$	AUROC 0.998, perplexity 6.2; default for the mini-project
$\delta =10$	detection saturates, but perplexity approaches 12: the text visibly degrades
smaller $\gamma$	more signal per green token, but likely tokens get excluded, so perplexity rises

Robust Variants of the PRF

Slide 24

The seed for the green/red split at position $i$ is a PRF of the previous $h$ tokens. Three ways to combine them (Kirchenbauer et al., “On the Reliability of Watermarks for Large Language Models”, ICLR 2024, §4.1):

Slide 24: Additive (sum the hash of the last h tokens), Skip (hash one chosen past token), Min (minimum over h hashes).

Slide 24: detection rate (left) and output diversity (right) vs. context width h. Min-SelfHash with h = 4 keeps detection high while raising diversity; Additive-LeftHash loses both as h grows. Source: KGW2, Fig. 4.

Intuition

A larger window $h$ makes the green list depend on more context, which is harder to learn for an attacker (see stealing below) and gives more varied text. But every edited token then destroys the green lists of $h$ following positions. “Min” is robust because changing one of the $h$ tokens often doesn’t change the minimum.

Alternatives: Aaronson’s Distortion-Free Gumbel Scheme

Slide 25

Watermarks must inject the signal somewhere in the sampling. KGW biases the distribution; the signal can also be put into the random draw of the sampler (Aaronson, 2022; Simons workshop slides, 2024; formalized by Kuditipudi, Thickstun, Hashimoto & Liang, “Robust Distortion-Free Watermarks”, 2023):

Gumbel-max watermark

$$r_v=\text{PRF}(\text{key},\text{ctx},v)\in (0,1)\forall v\in V$$ $$w_t=\arg \underset{v\in V}{\max }{r}_v^{1/p_v}\equiv \arg \underset{v}{\max }\frac{\log r_v}{\log p_v}$$

Detection sums a score over the emitted tokens:

$$S_T=\sum _{t=1}^T-\log (1-r_{w_t})\sim \text{Gamma}(T,1)\text{ under }{H}_0$$

With the watermark, $S_T$ is stochastically larger; test with a one-sided p-value against the Gamma distribution.

If the $r_v$ were truly random, $\arg {\max }_v{r}_v^{1/p_v}$ would pick token $v$ with probability exactly $p_v$: sampling is unchanged. But the chosen tokens tend to have large $r_v$, which the detector (knowing the key) can see.

	Aaronson	KGW
Marginal distribution	exactly $p$	shifted towards green
Statistic	$-\log (1-r_w)$	green count
Test	Gamma	binomial
Distortion	none (in expectation)	yes
Repeated context	repeats the same output	new random draw

Cryptographically Undetectable Watermarks

Slide 26

Undetectable watermark (Christ, Gunn & Zamir, COLT 2024)

A watermark is undetectable if no polynomial-time adversary with adaptive query access to the watermarked model can distinguish its outputs from the unwatermarked model with non-negligible advantage (paper).

One-bit construction: derive $u=F_k(\text{ctx})\in (0,1)$ from a PRF; use $u$ with the inverse CDF of the next-token distribution $p$ to read off one bit; embed only at positions with enough empirical entropy, and pass low-entropy positions through unchanged.

This is strictly stronger than Aaronson: undetectability holds on the joint output distribution, so the key can’t be extracted from polynomially many queries, and spoofing is provably blocked. The same idea exists for images (Gunn, Zhao & Song, ICLR 2025).

Scheme	Property
KGW	distortionary; can be resampled and combined with any sampler
Aaronson / Kuditipudi	distortion-free per token; leaks through repeated contexts
Christ-Gunn-Zamir	undetectable on the joint output; needs an entropy condition

Entropy is unavoidable

Zero-entropy positions (copy-paste, code with one obvious next token) can’t carry a hidden bit without changing the output. CGZ formalize this as a hard limit: low-entropy text can’t be watermarked.

Part IV: Attacks on Watermarks

Slide 27

What Could Happen to Watermarked Text in Daily Use?

Slide 28

Slide 28: machine paraphrasing, human paraphrasing and mixing with other text each lower the watermarked fraction, so more tokens are needed for detection.

Machine Paraphrasing and Paraphrases

Slides 29-30

Slide 29: paraphrasing with GPT-3.5 (copy-editor prompt) and with DIPPER (T5-XXL). Sources: Kirchenbauer et al., ICLR 2024; Krishna et al., DIPPER, NeurIPS 2023.

Tokens needed to reach $z=4$ (KGW2):

Text	Tokens needed
unparaphrased	~25
GPT-3.5 paraphrase	~150 to 300
DIPPER (T5-XXL)	~200 to 400
strong human paraphrase	~800 (at FPR = 10⁻⁵)

Slide 30: z-score after paraphrasing vs. tokens observed. Source: KGW2, Fig. 6.

Longer text helps

Paraphrasing lowers the signal per token, but it rarely removes it completely. Because $z$ grows with $\sqrt{T}$, the surviving green tokens still add up once the text is long enough.

Adversarial Attack Surfaces

Slide 31

Three attacks from and around Zhang, Edelman, Francati, Venturi, Ateniese & Barak, “Watermarks in the Sand”, ICML 2024:

Attack	How it works
Cipher	ask for the answer in base64, Chinese, or with “an emoji after every word”. The watermark is applied to the cipher text; after converting back, all n-grams are different, so the green lists no longer match
Oracle	a weak open model proposes answers, the watermarked strong model only judges them. Best-of-N approaches strong-model quality, with no watermark in the output
Copy-paste mixing	paste watermarked spans into a longer human document. CP-$k$-$X\%$ = $k$ spans that make up $X\%$ of the text. A single z-test over the whole text fails; the signal comes back with length

"Use 😍 after every word": stripping the emoji gives new n-grams.

Letter swap (n ↔ m, a ↔ e): harder to spot.

CP-k-X%: watermarked spans scattered in human text.

Watermark Identification and Spoofing

Slide 32

Watermark stealing queries a deployed watermarked model and statistically recovers the green-list signature (Jovanović, Staab & Vechev, ICML 2024; Pang et al., 2024). With the green lists known, the attacker can:

• Scrub: write AI text that avoids the green list, so the detector misses it.
• Spoof: write text that hits the green list and looks human-written, so the detector falsely accuses someone.

Slide 32: spoofing success vs. query budget for k = 1; about 10⁴ queries (~$50 of API spend) reach 80%+.

• Difficulty scales with $k$: the number of watermarked n-grams grows like $|V{|}^{k+1}$, but not all are needed.
• Spoofing risks: frame a model provider for text it did not produce; erode trust in the watermark detector itself.
• What blocks it: Christ-Gunn-Zamir undetectable schemes are robust against it by construction.

Part V: The Future of Provenance

Slide 33

Is Watermarking Worth It? The Volume Argument

Slide 34

Slide 34: 2021 ~99% of web text is human; 2026 machines dominate by volume and the blue slice is undetectable without watermarks; in a watermarked-by-default world only the gray slivers are unverifiable. Hover a slice for its share. Pie data: Naveen G. Rao's framing.

The volume argument

Without watermarking, everything machine-generated is hard to detect. With watermarking by default, all machine text is detectable, and only the attacked subset (paraphrased, removed) stays hard to detect, as before. Even a breakable watermark shrinks the unverifiable part enormously.

Unicode Watermarks

Slide 35

Whitemark replaces the ASCII space U+0020 with one of ~22 visually identical Unicode whitespace characters (U+2004 three-per-em, U+200B zero-width, …) and encodes one bit per whitespace. Variants for CJK glyph selectors and sub-pixel kerning exist (Variantmark, Printmark). Invisible, survives copy-paste (Sato, “Embarrassingly Simple Text Watermarks”, 2023). And it is defeated by one line:

re.sub(r"[​-‏︀-️ -]", "", text)  # strip the special whitespace

Slide 35: Whitemark. Source: Sato, 2023.

Unicode proposal L2/25-241 (Casper et al., Sep 2025) suggests two new code points, both looking like U+2060 WORD JOINER:

• TNCI (Training Non-Consent Indicator): authors mark text as not allowed for AI training.
• AGTI (AI-Generated Text Indicator): providers mark their output as machine-generated.

Default insertion: one per sentence at a random position; no bit encoding, no cryptographic binding. Status: earliest stage of review, not adopted.

The slide's point

If we want detection by default, and all watermarks are breakable anyway, why not use the simplest one?

The Reverse Approach: Proving Content Is Human

Slide 36

	C2PA / Content Credentials	Personhood credentials
What	a signed CBOR manifest that binds the capture device, edits and the AI generation tool to a hash chain; verification = X.509 certificate + hash check (C2PA spec 2.2)	offline issuance ceremony, online zero-knowledge proof of unique personhood (Adler, Hitzig, Jain, Birhane, Buterin et al., 2024)
Deployed	Adobe Firefly, DALL·E + Sora, Leica M11-P, Samsung Galaxy S25; the EU AI Act Art. 50 may make it mandatory in late 2026	extreme example: Worldcoin, an iris hash → Groth16 zk-SNARK (World ID) (privacy deep-dive; Buterin’s critique of biometric proof of personhood)
Side	provider side	user side

C2PA content credentials.

The Worldcoin orb.

Implemented correctly, zero-knowledge proofs can attest authorship or allowed attributes without leaking anything else. Composed, verified human + uncorrupted pipeline could guarantee attribution to humans: technically feasible, politically complicated.

Key Takeaways

Slide 37

#	Takeaway
1	Detection is hard, but often doable. Long, unedited LLM text is essentially detectable (LLMs write in a distinctive style); short, paraphrased or adversarially humanized text is hard.
2	Watermarking turns detection into a tractable hypothesis test. Red/green watermarks slightly change the output distribution; others replace the sampling procedure. Modern watermarks are ideally undetectable.
3	Attacks go far beyond paraphrasing: ciphers, oracles, copy-paste mixing, key stealing and spoofing.
4	Transparency could also come from verification. Watermarks shrink the undetectable machine text; provenance and personhood credentials grow the verifiably human text.

Summary

Method	Needs	Strength	Weakness
Word frequencies	a corpus	shows AI use at scale	useless for one short text
Perplexity	one LM	simple	prompts, memorized text
Binoculars	two related LMs	zero-shot, one threshold	short or edited text
Trained detector (Pangram)	lots of mirrored data	low false positives	humanizers, distribution shift
KGW watermark	the key, cooperation of the provider	p-values, cheap detection	paraphrase, cipher, stealing/spoofing
Aaronson / CGZ	the key	distortion-free / undetectable	needs entropy

Self-Test

Question cards (14)

What are the three facets of AI transparency, and why detect AI-generated text?

Answer

Model transparency (disclose weights, data, methods, evaluations), provenance tracking (signed chain of custody), synthetic-content detection (verifiable attribution as AI-generated). Detection serves authentication, observation of AI usage at scale, and separating human from machine data for future training.

How does perplexity-based detection work, and why does it fail?

Answer

Text generated by a model (or a related one) has low perplexity under it, human text normal perplexity. It fails because perplexity depends on the missing prompt (an unusual prompt yields high-perplexity machine text) and because memorized human text has low perplexity for all models.

Binoculars uses an observer and a performer. Why does it compare two perplexities instead of thresholding one?

Answer

Neither perplexity alone separates human from machine text, since it varies with prompt and topic (AUC 0.66 and 0.71). The ratio B = log PPL / log X-PPL normalizes by how surprising the text is for a language model in general, cancelling the prompt- and topic-driven part; model-written text then shows an anomalously low ratio (AUC 0.99), human text does not.

How are strong trained detectors like Pangram built?

Answer

For each human document, generate a synthetic mirror that matches topic, length and register. Train, find the human texts that are falsely flagged, generate mirrors for them and retrain, until it plateaus; include known humanizers. This iterated hard-negative mining keeps the false-positive rate low.

How does the KGW watermark change generation?

Answer

A PRF of a secret key and the previous k tokens splits the vocabulary into a green list (fraction γ) and a red list. δ is added to the green logits before the softmax, so green tokens become more likely without being forced; where the model is confident the choice barely changes.

How is a KGW watermark detected? Give the formula.

Answer

Recompute the green lists with the key, count green tokens |s|_G among T, and compute z = (|s|_G − γT)/√(Tγ(1 − γ)). A large z (e.g. > 4) means far more green tokens than expected without watermark; the p-value comes from the normal tail, or the exact binomial for short texts. No model access is needed.

What do δ and γ trade off?

Answer

A larger δ gives stronger detection but lower quality: δ = 2, γ = 0.5 reaches AUROC 0.998 at perplexity 6.2, δ = 10 saturates detection but pushes perplexity to ~12. A smaller γ gives more signal per green token but excludes likely tokens more often, which raises perplexity.

Compare KGW, Aaronson's Gumbel scheme and Christ-Gunn-Zamir watermarks.

Answer

KGW biases the distribution towards green tokens and tests the green count (binomial); it distorts the output. Aaronson picks argmax r_v^{1/p_v} with PRF values r_v, so each token keeps its exact probability, and tests a Gamma-distributed score; repeated contexts give repeated outputs. CGZ embeds bits via PRF and inverse CDF only at high-entropy positions and is cryptographically undetectable on the joint output, which also blocks spoofing.

Why is entropy unavoidable for watermarking?

Answer

A watermark must influence a choice. At zero-entropy positions (copy-paste, code with one obvious next token) there is no choice to influence without changing the output, so no hidden bit fits there. Low-entropy text can’t be reliably watermarked.

How does paraphrasing affect a watermark, and why does detection still work for long texts?

Answer

Paraphrasing replaces many tokens, so fewer are green and the signal per token drops: z = 4 needs ~25 tokens unparaphrased, ~150 to 300 after GPT-3.5, ~800 after a strong human paraphrase. Some green tokens survive, and since z grows like √T, long enough text is still detected.

Explain the cipher, oracle and copy-paste attacks.

Answer

Cipher: the model writes in a transformed form (base64, emoji after every word); converting back changes all n-grams, so the green lists no longer match. Oracle: a weak open model generates, the watermarked model only judges, so the output carries no watermark. Copy-paste: watermarked spans are diluted in human text, so one global z-test fails, though the signal returns with length.

What is watermark stealing, and what is the difference between scrubbing and spoofing?

Answer

Querying a watermarked model (~10⁴ queries, ~$50 for k = 1) lets an attacker estimate the green lists statistically. Scrubbing then avoids green tokens so AI text goes undetected; spoofing uses green tokens so human-looking text is falsely flagged as model output, framing the provider or undermining trust in the detector.

What is the volume argument, and why use even a simple Unicode watermark?

Answer

As machine text comes to dominate the web, without watermarks all of it is hard to detect. With watermarking by default, only the attacked subset stays hard to detect, as before. So even a breakable mark (Unicode whitespace, one regex removes it) makes most machine text detectable by default.

How do C2PA and personhood credentials reverse the detection problem?

Answer

Instead of detecting AI text, they verify human content. C2PA binds capture device, edits and AI tools to a signed hash chain (provider side); personhood credentials prove unique human authorship with zero-knowledge proofs (user side). Together they could attribute content to verified humans: technically feasible, politically complicated.

Multiple Choice

Multiple choice (5)

γ = 0.5, T = 100 tokens, 70 of them green. What is the z-score?

• 2

• 4

• 7

• 20

Explanation

z = (70 − 50)/√(100 · 0.5 · 0.5) = 20/5 = 4, a clear detection.

Why does Binoculars divide by the cross-perplexity?

• To make the detector faster.

• To cancel the prompt- and topic-driven part of perplexity.

• To add a watermark to the text.

• Because the observer model has no perplexity.

Explanation

Perplexity alone mixes “written by a model” with “surprising content”. The cross-perplexity measures the expected surprise for this content, so the ratio isolates the first part.

Which property does the Christ-Gunn-Zamir watermark have that KGW lacks?

• It works on zero-entropy text.

• It survives any paraphrase.

• Its outputs are computationally indistinguishable from unwatermarked outputs, which blocks spoofing.

• It needs no secret key.

Explanation

CGZ is undetectable on the joint output distribution, so the key can’t be learned from queries. It still needs entropy and a key, and heavy paraphrasing still hurts.

An attacker writes human-looking text full of green tokens so that the detector blames the model provider. What is this?

• scrubbing

• spoofing

• a cipher attack

• an oracle attack

Explanation

Scrubbing hides AI text from the detector; spoofing makes the detector fire on text the model did not write. Both need a stolen green-list signature.

What happens to KGW detection when δ is raised from 2 to 10?

• Detection gets weaker and quality better.

• Detection saturates, but perplexity rises to ~12 and the text degrades.

• Nothing changes, only γ matters.

• The watermark becomes undetectable.

Explanation

δ = 2, γ = 0.5 already gives AUROC 0.998 at perplexity 6.2; more bias mostly costs quality.

References

All sources cited on the slides, in slide order (30 entries)

Slide	Source	Key point
4	NIST AI 100-4, 2024	three facets of transparency
6	Kulveit et al., “Gradual Disempowerment”, 2025	why attribution matters
6	Liang et al., 2024	measuring LLM usage at scale
7	EU AI Act, Art. 50	marking requirement
7	EU Code of Practice draft, Dec 2025	metadata + watermark + disclosure
7	California SB 942	AI transparency act (no text)
7	China CAC labeling rules	labeling of synthetic content
11	Liang et al., Nat. Hum. Behav. 2025	LLM word signatures in papers
11	Liang et al., “Why Does ChatGPT ‘Delve’ So Much?”, COLING 2025	origin of marker words
13	Hans et al., “Spotting LLMs With Binoculars”, ICML 2024	Binoculars score (not linked on the slide)
14	Emi & Spero, Pangram technical report, 2024	trained detector with mirrors
14	Masrour, Emi & Spero, DAMAGE, COLING 2025	detectors vs. humanizers
14	Pangram ICLR 2026 study	21% of reviews AI-generated
18, 20, 21, 23	Kirchenbauer et al., “A Watermark for Large Language Models”, ICML 2023	KGW watermark, z-test, δ/γ trade-off
24, 29, 30	Kirchenbauer et al., “On the Reliability of Watermarks for Large Language Models”, ICLR 2024	seeding schemes, paraphrase robustness
25	Aaronson, “My AI Safety Lecture for UT Effective Altruism”, 2022	Gumbel-max watermark
25	Aaronson, Simons LLM workshop slides, 2024	Gumbel-max watermark
25	Kuditipudi et al., “Robust Distortion-Free Watermarks”, 2023	formalization
26	Christ, Gunn & Zamir, “Undetectable Watermarks for Language Models”, COLT 2024	undetectability
26	Gunn, Zhao & Song, “An Undetectable Watermark for Generative Image Models”, ICLR 2025	undetectable image watermarks
29, 30	Krishna et al., DIPPER, NeurIPS 2023	paraphrasing evades detectors
31	Zhang et al., “Watermarks in the Sand”, ICML 2024	oracle attack, impossibility
32	Jovanović, Staab & Vechev, “Watermark Stealing”, ICML 2024	stealing, scrubbing, spoofing
32	Pang et al., 2024	watermark identification
35	Sato, “Embarrassingly Simple Text Watermarks”, 2023	Unicode whitespace watermarks
35	Casper et al., Unicode proposal L2/25-241, 2025	TNCI and AGTI code points
36	C2PA specification 2.2	content credentials
36	Adler et al., “Personhood Credentials”, 2024	proof of personhood
36	World / Worldcoin privacy deep-dive	World ID
36	Buterin, biometric proof-of-personhood critique, 2023	risks of biometric PoP

Related

• Previous: Lecture 4: Open-Weight Safety · Next: Lecture 6: Data Privacy and Memorization · Course: Overview
• Exam and reference: Exam Structure · Study Plan · Formula Sheet · Glossary
• Concepts: Watermarking, AI Text Detection, Content Provenance, Decoding and Sampling
• The watermark was first shown in Lecture 1; sampling and logits in Lecture 2.