Loss and Risk

Definition

• A loss function $\ell :\mathcal{X}\times \mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}_{\ge 0}$ says how expensive it is to predict $y^{\prime }$ when the truth is $y$. 0-1 loss: $1_{y\ne y^{\prime }}$. Squared loss: $(y-y^{\prime }{)}^2$.
• The true risk of $f:\mathcal{X}\to \mathcal{Y}$ is its expected loss under the data distribution $P$.

Formula

$$R(f):=E(\ell (X,Y,f(X))),\text{0-1 loss: }R(f)=P(f(X)\ne Y)$$

The goal of machine learning is to use training data to construct a function $f_n$ whose true risk is as small as possible. The smallest possible value is the Bayes risk $R^{\ast }$. The loss may depend on $x$ and on the type of error (asymmetric costs), and its choice influences the inductive bias.

Appears in

• Lecture 1.1, setup and loss functions
• Lecture 1.1, definition of the true risk
• Lecture 1.1, conditional risk with costs
• Lecture 1.2, the empirical risk as an estimate of the true risk

Related concepts

• Bayes Classifier
• Surrogate Loss
• Empirical Risk Minimization