Past exam of the mathematics course of the University of Cambridge / 2022 / iii / Paper 218 / 3

For word-count vector $x$, the fitted logistic model isHolding other counts fixed, one additional occurrence of "dollar" multiplies the fitted spam odds by $e^{1.859318}$.

The logistic classifier isThe coefficients maximize the Bernoulli logistic-regression model likelihood over the training emails.

The decision boundary is the hyperplaneAt threshold $q$ the classifier predicts spam whenso $q=1/2$ gives the original classifier. Varying $q$ translates the boundary parallel to itself: raising $q$ shrinks the region classified as spam and generally trades fewer false positives for more false negatives.

Writing ${\eta }_i={\beta }_0+x_i^T\beta$, the log-likelihood isWith 500 observations and 5000 word predictors, the augmented design matrix cannot have full column rank. There are nonzero coefficient directions that leave every ${\eta }_i$ unchanged, so any maximizer belongs to an affine family and is not unique. In addition, the high-dimensional features may completely separate the classes, in which case no finite maximizer exists at all.

Principal component analysis centers the word-count vectors, diagonalizes their sample covariance matrix, and projects onto the eigenvectors with the largest eigenvalues. Fitting logistic regression to $d<n$ principal-component scores removes exact collinearity and yields a lower-dimensional design.

Each principal component is an unsupervised linear combination of many words chosen to explain predictor variance; it is not a word selected for association with spam. The dimension can be chosen from a scree plot or cumulative explained variance, but cross-validating the downstream classification loss better targets prediction.

For fixed $A$, differentiating under the constraint ${\sum }_i{z}_i=0$ givesBecause the columns $u_j$ are orthogonal,The fitted value is therefore $\overline{X}+{\Pi }_A(X_i-\overline{X})$, where ${\Pi }_A$ is the orthogonal projection onto the column space of $A$.

The residual sum of squares is minimized by choosing this column space to be the span of the $d$ leading eigenvectors ofThus $\hat{A}$ may be taken to have those orthonormal eigenvectors as columns. Their nonzero scales are immaterial because inverse scaling of the scores leaves $A{z}_i$ unchanged.