Definition
At a fixed point , with the expectation over the random training sample: the variance is and the bias is , where is the regression function.
Formula
- Proof: add and subtract ; the cross term is 0 because is deterministic and .
- Variance ~ estimation error (depends on the data and the capacity of ), bias ~ approximation error (depends only on the capacity).
- Classical picture: bias falls and variance rises with model complexity, the test error is U-shaped. Shrinking predictions trades a little bias for less variance (regularization, Lecture 5).