Benign Overfitting

Definition

A learning method overfits benignly if it reaches training error 0 on noisy data and its excess risk is still small (goes to 0).

Formula

True function 0, $x_i\sim \mathcal{N}(0,I_d)$, $y_i\sim \mathcal{N}(0,{\sigma }^2)$: the minimum norm interpolator has risk ${\sigma }^2\frac{n}{d-n-1}$. Spikes: $P(x\in {\bigcup }_i{S}_i(t))\le 2n\exp (-cd{t}^2)$.

With a real signal and isotropic inputs the bias $\Vert {\theta }^{\ast }{\Vert }^2(d-n)/d$ stays: not benign. Benign overfitting needs an aligned covariance: few large eigenvalues that carry the signal and many small, flat tail eigenvalues that interpolate the noise (Theorem 5: excess risk for a general covariance, regime 3).

Appears in

• Lecture 5, first mention for d much larger than n
• Lecture 8, Theorem 3 (benign overfitting in the toy setup), spiky-smooth structure, Theorem 5 (excess risk for a general covariance), four regimes

Related concepts

• Implicit Regularization
• Double Descent
• Uniform Convergence