Past exam of the mathematics course of the University of Cambridge / 2023 / ii / Paper 3 / 28K (Principles of Statistics)

Let $f_i$ be the multivariate normal density of $N_p({\mu }_i,{\Sigma }_i)$. Under zero-one loss, the Bayes classifier chooses the class with the larger posterior probability. ThusEquivalently, it chooses class one when the log posterior oddsis nonnegative. The decision boundary is $D_{{\pi }_0}(x)=0$.

If ${\Sigma }_0={\Sigma }_1=\Sigma$, the $x^T{\Sigma }^{-1}x$ terms cancel, leavingan affine function. This is the linear boundary of linear discriminant analysis. If ${\Sigma }_0\ne {\Sigma }_1$, the quadratic part iswhich is nonzero, so the boundary is a possibly degenerate quadratic hypersurface, as in quadratic discriminant analysis. This is the Gaussian Bayes classifier.

Condition on the observed value $X=x$. If a classifier reports class zero, its conditional probability of error is $\mathbb{P}(Y=1\mid X=x)$; if it reports class one, the conditional error is $\mathbb{P}(Y=0\mid X=x)$. Therefore the smaller conditional error is attained by class one exactly whenand by class zero exactly when the reverse strict inequality holds. Integrating these pointwise conditional errors proves that ${\delta }_{{\pi }_0}$ is a Bayes classifier.

The only possible nonuniqueness lies on the tie setBecause the two Gaussian class distributions are distinct, $D_{{\pi }_0}$ is not identically zero. It is a nonzero polynomial of degree at most two, so its zero set has Lebesgue measure zero. Both Gaussian laws have densities with respect to Lebesgue measure, hence assign probability zero to $B$. Every Bayes rule must therefore agree with ${\delta }_{{\pi }_0}$ almost surely. This proves the uniqueness of a Bayes classifier up to the standard null-set equivalence of decision rules.

For $0<\pi <1$, let ${\delta }_{\pi }$ be the Bayes classifier with class-zero prior $\pi$, and define its two class-conditional error probabilities byWriting the likelihood ratio as $L=f_1/f_0$, the rule chooses class one whenThe Gaussian likelihood-ratio level sets have probability zero, so dominated convergence shows that $e_0$ and $e_1$ depend continuously on $\pi$. As $\pi \downarrow 0$, the threshold tends to zero and the classifier chooses class one almost surely, givingAs $\pi \uparrow 1$, it chooses class zero almost surely, giving the opposite limit $(0,1)$. The intermediate value theorem therefore provides ${\pi }^{\ast }\in (0,1)$ such that

The rule ${\delta }_{{\pi }^{\ast }}$ is an equalizer rule with worst-case risk $r$. For any classifier $\delta$,because ${\delta }_{{\pi }^{\ast }}$ minimizes the integrated risk for its prior. Hence ${\delta }_{{\pi }^{\ast }}$ is a minimax decision rule, as summarized by the minimax Gaussian Bayes classifier from equal class errors.

The prior is indeed least favorable. Its Bayes risk is $r$, while for any other prior $q$ the Bayes risk is at most the integrated risk of the same equalizer rule ${\delta }_{{\pi }^{\ast }}$, namelyThus no prior has larger Bayes risk.