Bayes Classifier

Definition

The Bayes risk is $R^{\ast }={\inf }_{f}R(f)$ over all measurable $f$. A function attaining it is a Bayes classifier (Bayes predictor) $f^{\ast }$; it need not be unique. The regression function is $\eta (x)=E(Y\mid X=x)$, for labels $\{0,1\}$ equal to $P(Y=1\mid X=x)$.

Formula

0-1 loss:

$$f^{\ast }(x)=1_{\eta (x)\ge 1/2},R^{\ast }=E[\min (\eta (X),1-\eta (X))]$$

Squared loss (regression):

$$f^{\ast }(x)=\eta (x),R^{\ast }=E[\mathrm{Var}(Y\mid X)]$$

The Bayes classifier needs the unknown distribution, so it is a benchmark, not an algorithm. $P$ is determined by the marginal $\mu$ of $X$ and by $\eta$ (Proposition 1). Consistency asks whether the risk of a learning algorithm converges to $R^{\ast }$.

Appears in

• Lecture 1.1, definition
• Lecture 1.1, regression function and Proposition 1
• Lecture 1.1, Theorem 2 and the Bayes error
• Lecture 1.1, Theorem 4 for the squared loss
• Lecture 1.2, the goal of learning: R(f_n) close to R*
• Lecture 1.2, the plug-in classifier

Related concepts

• Bayesian Decision Theory
• Surrogate Loss
• Loss and Risk