Past exam of the mathematics course of the University of Cambridge / 2026 / iii / Paper 218 (Statistical Learning in Practice)

Expanding the exponent of the Inverse Gaussian distribution givesThus the exponential dispersion family representationhasandThe standard cumulant identities yieldHence the variance function is $V(\mu )={\mu }^3$, and the canonical link function is $g(\mu )=\theta (\mu )=-1/(2{\mu }^2)$.

Writing $d_i$ for difficulty, model1 assumes independent responseswith common dispersion. The estimates are ${\hat{\beta }}_0=9.0827$ and ${\hat{\beta }}_1=-1.4323$.

One extra difficulty level decreases the fitted inverse squared mean response time by $1.4323$. Since $\mu =({\beta }_0+{\beta }_{1}d{)}^{-1/2}$, this means that fitted mean response time increases with difficulty. The negative coefficient is therefore unsurprising; its sign looks counterintuitive only if the inverse-squared link function is ignored. The independence assumption is questionable because every subject contributes eight repeated responses.

The Pearson residual iswhere the moment estimate of dispersion isIf the fitted model is adequate and the deviance residual sum is close to the Pearson statistic, then

For response $j$ from subject $s$, model2 is the generalized linear mixed modelwith conditional independence given the random intercepts. The fitted values are ${\hat{\beta }}_0=8.9966$, ${\hat{\beta }}_1=-1.2317$, ${\hat{\tau }}^2=1.7533$, and fitted residual dispersion $0.7591$.

The random intercept models persistent between-subject differences and the resulting within-subject dependence among repeated measurements. That is the main feature absent from model1.

Testwith a likelihood-ratio test. Model1 has three likelihood parameters and model2 has four, so their AIC values giveThe statistic isUsing a naive ${\chi }_1^2$ reference gives $p=0.0341$. Because the null variance lies on the boundary, the standard asymptotic reference is the mixture $\frac{1}{2}{\chi }_0^2+\frac{1}{2}{\chi }_1^2$, giving $p=0.0170$. Either calibration rejects at the five-percent level and supports a subject random effect.

Order the observations so thatusing a fixed or randomized rule for ties. The L-nearest-neighbour classifier estimatesand predicts a class maximizing ${\hat{p}}_k(x)$.

Conditional on $X_1,\dots ,X_n$, the $L$ selected class indicators are independent Bernoulli variables. ThereforeandTaking expectations proves the claim.

Let $N_{\delta }$ count sample points in the intersection of $[0,1{]}^d$ with the ${\ell }^{\infty }$ ball of radius $\delta$ around $x$. That intersection has volume at least ${\delta }^d$, so $N_{\delta }\sim \mathrm{Bin}(n,q)$ with $q\ge M{\delta }^d$. The event $\Vert X_{(L)}-x{\Vert }_{\infty }>\delta$ implies $N_{\delta }<L$. Since $\mathrm{Var}(N_{\delta })\le nq$ and $nq\ge nM{\delta }^d\ge L$, Chebyshev inequality givesTaking the minimum with the trivial bound one proves the result.

Put $R=\Vert X_{(L)}-x{\Vert }_{\infty }$ and ${\delta }_0=(2L/(nM){)}^{1/d}$. If ${\delta }_0\ge 1$, the claim follows from $R\le 1$. Otherwise, for $\delta \ge {\delta }_0$, part (c) givesThe tail-sum formula then yields

The Lipschitz continuity of $p_1$ givesPart (d) therefore implies

By the Cauchy-Schwarz inequality, part (b), and part (e),This bound tends uniformly to zero ifThe plug-in classifier excess-risk bound then shows that the misclassification risk of the nearest-neighbour classifier converges to the Bayes risk.

The ${\xi }_i$ are slack variables of a support vector machine. The solid line is the support-vector-machine decision boundaryThe dashed lines are the two support-vector-machine margin boundariesEach is at perpendicular distance $1/\Vert \hat{\beta }{\Vert }_2$ from the decision boundary, so the full margin width is $2/\Vert \hat{\beta }{\Vert }_2$.

For a monarch point, ${\hat{\xi }}_i=\max \{0,1-(\hat{\alpha }+X_i^{\top }\hat{\beta })\}$. Point P2 lies well beyond the monarch-side dashed margin, so a plausible value is $0$. P1 lies between the decision boundary and that margin, so a plausible value is about $0.4$. P3 lies on the wrong side of the decision boundary, so its slack exceeds one; about $1.3$ is plausible. Points P1 and P3 are support vectors, while P2 is not.

Increasing $C$ raises the cost of slack variables of a support vector machine. The fit therefore generally accepts fewer margin violations and misclassifications, at the price of a larger $\Vert \hat{\beta }{\Vert }_2$ and hence a narrower support-vector-machine margin. Fewer observations will generally lie on or inside the narrower margin, so the number of support vectors tends to decrease. These are qualitative tendencies; individual counts need not vary monotonically for every data set.

A training observation is misclassified only ifThe SVM constraint then forces ${\hat{\xi }}_i\ge 1$. HenceSumming and dividing by $n$ proves ${\widehat{\mathrm{Err}}}_{\mathrm{tr}}\le n^{-1}{\sum }_i{\hat{\xi }}_i$.

The code computes Leave-one-out cross-validation. Ifthen observation $i$ is not a support vector. Removing it leaves the optimum unchanged, and the resulting classifier still classifies it correctly. A leave-one-out error can therefore occur only for an observation on or inside the margin, which provesThus the fraction of training observations on or inside the margin is an upper bound on leave-one-out error. One can refit only after deleting support vectors, reusing the full fit for every other observation. Alternatively, K-fold cross-validation needs only $K$ fits and is often preferable for larger data sets.

A real positive-semidefinite kernel is a symmetric function $k:X\times X\to \mathbb{R}$ such that every finite Gram matrix $(k(x_i,x_j))$ is positive semidefinite. The Moore-Aronszajn theorem says that there are a Hilbert space $H$ and a feature map $\varphi :X\to H$ such thatEquivalently, $H$ can be chosen as the unique Reproducing kernel Hilbert space with reproducing kernel $k$.

Let $(e_{r,a},e_{r,b}{)}_{r=1}^L$ be the standard orthonormal basis of ${\mathbb{R}}^{2L}$ and defineThenThus $g$ is a positive-semidefinite kernel.

The empirical kernel covariance operator $\Sigma$ is self-adjoint and positive semidefinite. Maximize $⟨u,\Sigma u⟩$ subject to $⟨u,u⟩=1$. The first variation of the Lagrange multiplier functional givesso $\Sigma v=\lambda v$. Taking the inner product with $v$ gives

Positive definiteness of $K$ makes the vectors $\varphi (x_1),\dots ,\varphi (x_n)$ linearly independent. In particular $\Sigma \ne 0$, so the maximum Rayleigh quotient is positive and $\lambda >0$. The eigenvector equation giveswhich lies in their span. Hence $v={\sum }_i{\alpha }_i\varphi (x_i)$ for some $\alpha \in {\mathbb{R}}^n$.

Substituting $v={\sum }_j{\alpha }_j\varphi (x_j)$ into $\Sigma v=\lambda v$ givesLinear independence yields $K\alpha =n\lambda \alpha$. The unit-norm constraint gives

The feature space may be extremely high-dimensional or infinite-dimensional, and $\varphi$ may be known only implicitly. Instead, compute the leading eigenvector $\alpha$ of the kernel matrix $K$, normalize it by ${\alpha }^{\top }K\alpha =1$, and use the kernel trick:This requires only kernel evaluations.

Kernel principal component analysis can represent nonlinear low-dimensional structure by performing linear PCA in a nonlinear feature space. It can also work directly with structured objects such as strings through a kernel, without assigning them explicit finite-dimensional coordinates. Both capabilities are unavailable to ordinary linear PCA on the original variables.

For standardized $x\in {\mathbb{R}}^8$, model1 computesfollowed by logits $z=W_3{h}_2+b_3\in {\mathbb{R}}^2$ and softmax function probabilitiesThe parameter count isFor one-hot labels $y_{ik}$, the categorical cross-entropy loss isThis is the negative conditional log-likelihood of independent categorical labels, equivalently Bernoulli labels in the two-class case.

Stochastic gradient descent replaces the full empirical-loss gradient by the gradient on a randomly ordered observation or mini-batch, then updates $\theta \leftarrow \theta -\eta \hat{\nabla }L(\theta )$. The training half contains $768/2=384$ observations, so batches of 16 give $384/16=24$ updates per epoch. Over 100 epochs every parameter is updated $2400$ times.

The neural-network likelihood is nonconvex, so optimization can stop at a local optimum or saddle rather than a global maximum. One hundred epochs may be insufficient for convergence. Mini-batch gradient noise together with a fixed positive learning rate can keep the iterates fluctuating around a stationary point rather than reaching it exactly. Any of these prevents the final parameters from being exact maximum-likelihood estimators.

The dashed curve is training accuracy: it continues to rise as optimization adapts to the training observations. The solid curve is testing accuracy: it peaks near 10 epochs and then declines. The widening gap is overfitting.

A sensible choice is about 10 epochs, selected by early stopping at the maximum validation accuracy. In a proper analysis, a validation set rather than the final test set should choose this epoch. Early stopping is an implicit regularization method because it limits how far the parameters can adapt to training-specific noise.