Bayesian Decision Theory

Definition

The Bayes decision rule predicts the label with the smallest conditional risk $R(\hat{y}\mid X=x)={\sum }_y\ell (x,y,\hat{y})P(Y=y\mid X=x)$. Special cases: MAP (0-1 loss, largest posterior) and maximum likelihood (additionally a uniform prior, largest $p(x\mid y)$).

Formula

Two classes, ${\ell }_{01}$ = cost of predicting 0 when $Y=1$, ${\ell }_{10}$ = cost of predicting 1 when $Y=0$:

$$\text{predict 1}⟺\eta (x)>\frac{{\ell }_{10}}{{\ell }_{10}+{\ell }_{01}}⟺{\pi }_1{\ell }_{01}p(x\mid 1)>{\pi }_0{\ell }_{10}p(x\mid 0)$$

Gaussians with equal variance:

$$x^{\ast }=\frac{{\mu }_0+{\mu }_1}{2}+\frac{{\sigma }^2}{{\mu }_1-{\mu }_0}\ln \frac{{\pi }_0{\ell }_{10}}{{\pi }_1{\ell }_{01}}$$

Decision boundaries are where the two weighted curves cross. A larger prior or a larger miss cost for a class enlarges its region. Given a boundary, plugging it into the equality gives the prior or the cost ratio backwards.

Appears in

• Lecture 1.1, the four approaches
• Lecture 1.1, the Bayes decision rule
• Lecture 1.1, simulations and the risk of a threshold
• Lecture 1.1, exam training with boxes, triangles and Gaussians

Related concepts

• Bayes Classifier
• Loss and Risk