Memorization

Definition

• $k$-extractable: a length-$k$ prefix from the training set makes the model output $y$ verbatim.
• Adversarial compression ratio: $\text{ACR}(M,y)=|y|/|x^{\ast }|$ with $x^{\ast }$ the shortest prompt that elicits $y$; memorized iff ACR > 1.

• Why: long-tailed data makes memorization necessary (Feldman); capacity ≈ 2 bits per parameter (Allen-Zhu & Li).
• Scaling: log-linear in model size (+19 pp per 10×), duplication and prompt length; deduplication cuts verbatim output 10×.
• Attacks: GPT-2 extraction (generate, rank by relative loss, verify); ChatGPT divergence attack (“repeat poem forever”) with 150× more memorized output.
• Memorization ≠ copying: a distribution property vs. a legal test on the output.

Appears in

• Lecture 6, definitions
• Lecture 6, why models memorize
• Lecture 6, duplicates
• Lecture 6, extraction attacks
• Lecture 6, ChatGPT divergence attack
• Lecture 6, memorization vs. copying

Related concepts

• Privacy Threat Models
• Data Poisoning