Double Descent

Definition

Plot the test error against the model size. In the under-parameterized regime (fewer parameters than data points) it follows the classical U-curve. At the interpolation threshold (training error 0 becomes possible, roughly parameters = data points) it peaks. In the over-parameterized regime it can decrease again, sometimes below the classical optimum.

Formula

Thresholds: random features $m=n$; polynomial of degree $d$: $d+1=n$; network with $K$ outputs: about $nK$ parameters. Isotropic linear model: excess risk ${\sigma }^{2}d/(n-d-1)$ below, ${\sigma }^{2}n/(d-n-1)+\Vert {\theta }^{\ast }{\Vert }^2(d-n)/d$ above the threshold, infinite at $d\approx n$.

Double descent does not always happen (polynomial regression with monomials gets worse beyond the threshold), single and multiple descent also occur, and the shape depends on how parameters are counted. The peak comes from the exploding variance of barely interpolating solutions; the second descent needs implicit regularization and a benign structure.

Appears in

• Lecture 1.2, the modern regime
• Lecture 8, random feature example, case without double descent, Theorem 4 (excess risk in both regimes)

Related concepts

• Bias-Variance Decomposition
• Benign Overfitting
• Random Features