Past exam of the mathematics course of the University of Cambridge / 2022 / iii / Paper 218 / 2

First recode the labels as $y_i=2Y_i-1\in \{-1,1\}$; with labels $0$ and $1$, the displayed constraints for class zero can never hold. Add an unpenalized intercept $b$ so that the separating hyperplane need not pass through the origin, and solveCentering and scaling predictor columns is also advisable because the Euclidean penalty depends on their units.

The support vector machine classifier isUnder the constraints, the two classes lie beyond the parallel support-vector-machine margin boundaries $b+x^T\beta =\pm 1$. Their distance is $2/\Vert \beta {\Vert }_2$, so minimizing the norm maximizes the geometric margin. The observations touching the margin are the support vectors.

The hard-margin constraints are feasible exactly when the classes are separated by a separating hyperplane. Failure after the corrections in part a therefore means the classes overlap.

Introduce slack variables of a support vector machine and solve the soft-margin problemsubject toLarge $C$ strongly penalizes violations and approaches the hard-margin solution when separation is possible. Small $C$ tolerates more violations in exchange for a wider, more strongly regularized margin.

The kernel trick replaces inner products $\varphi (x_i{)}^T\varphi (x_j)$ by evaluations of a positive-semidefinite kernel $k(x_i,x_j)$ without explicitly constructing the feature vectors. In the corresponding reproducing-kernel Hilbert space, the hard-margin problem isEquivalently, its dual isThere is no upper bound ${\alpha }_i\le C$ because this is the hard-margin, rather than soft-margin, problem.

The successful hard-margin fit proves that the transformed observations are separable. This creates complete separation in unpenalized logistic regression: scaling a separating coefficient vector continually raises the likelihood, so no finite maximum-likelihood estimate exists. The nearly singular observed information produces the enormous reported standard errors.

A ridge-penalized logistic regression, or equivalently a Bayesian logistic model with a proper Gaussian prior, gives finite, stable coefficients. Firth bias reduction is another standard remedy.