Surrogate Loss

Definition

For labels $\pm 1$, a scoring function $f:\mathcal{X}\to \mathbb{R}$ classifies by $\mathrm{sign}(f(x))$; $(x,y)$ is classified correctly iff the margin $yf(x)>0$. Surrogate losses replace the non-convex 0-1 loss by convex functions of the margin.

Formula

Bayes predictors, with ${\eta }_1(x)=P(Y=1\mid X=x)-P(Y=-1\mid X=x)$:

$$\text{hinge: }\mathrm{sign}({\eta }_1),\text{squared: }{\eta }_1,\text{exponential: }\frac{1}{2}\log \frac{1+{\eta }_1}{1-{\eta }_1},\text{logistic: }\log \frac{1+{\eta }_1}{1-{\eta }_1}$$

The Bayes predictors differ (different inductive biases), but all have the sign of ${\eta }_1$: the losses are classification-calibrated. The exponential loss is the loss that AdaBoost minimizes (Lecture 6).

Appears in

• Lecture 1.1, scoring functions
• Lecture 1.1, table of Bayes predictors and proof for the exponential loss
• Lecture 1.2, convex relaxations because 0-1 ERM is NP hard

Related concepts

• Bayes Classifier
• Loss and Risk