AI Text Detection

Definition

AI text detection asks: did a language model write this text? (Not: is it true? Not: is this image fake?)

Method	Idea	Problem
Word frequencies	marker words (“delve”) spike after 2022	only at corpus scale
Perplexity	model text has low PPL	prompt missing; memorized human text
Binoculars	$B=\log {\text{PPL}}_{M_1}/\log \text{X-PPL}$ with observer and performer model	short or edited text
Trained classifiers	synthetic mirrors + hard-negative mining (Pangram)	humanizers, distribution shift

Binoculars works because the cross-perplexity normalizes by how surprising the content is for a language model in general, cancelling the prompt and topic (AUC 0.66 and 0.71 alone, 0.99 as ratio).

Appears in

• Lecture 5, perplexity baseline
• Lecture 5, Binoculars
• Lecture 5, trained detectors

Related concepts

• Watermarking
• Content Provenance