Past exam of the mathematics course of the University of Cambridge / 2024 / iii / Paper 218 (Statistical Learning in Practice)

Write $Y_i$ for the colony count and $d_i$ for the dose on plate $i$. The first fit is the Poisson regressionThus its log-likelihood isThe Poisson deviance relative to the saturated model iswhere a summand with $y_i=0$ uses $0\log 0=0$.

At dose zero the fitted expected value is $e^{{\hat{\beta }}_0}=e^{3.321995}\simeq 27.72$ eradicated colonies. Increasing dose by one unit multiplies the fitted mean by $e^{{\hat{\beta }}_1}=e^{0.0001901}\simeq 1.000190$; for example, an increase of $100$ units multiplies it by about $1.0192$. The positive fitted effect is small and its displayed two-sided $p$-value, $0.105$, gives little evidence against a zero dose coefficient.

For the Poisson variance function $V(\mu )=\mu$, the code computes the Pearson chi-squared statisticand the Pearson dispersion estimatorThe first quantity measures goodness of fit; the second estimates the dispersion parameter, which equals one in a correctly specified Poisson regression. A standard rough calculation substitutes the residual deviance for the Pearson statistic and givesIf the reported upper-tail probability $4.908651\times 10^{-11}$ is inverted numerically, the actual Pearson statistic used by the code is about $82.93$, giving $\hat{\varphi }\simeq 5.18$. Either calculation reveals severe overdispersion.

The Pearson goodness-of-fit test has null hypothesis that the independent counts follow the fitted Poisson regression, in particular $\mathrm{Var}(Y_i\mid d_i)={\mu }_i$, against the alternative that the model does not fit; in this setting the scientifically relevant direction is overdispersion, $\mathrm{Var}(Y_i\mid d_i)>{\mu }_i$. Under the null, $X_P^2$ is approximately chi-squared distribution with $16$ degrees of freedom. Its tiny $p$-value decisively rejects the Poisson variance assumption.

The negative binomial regression keeps the logarithmic mean model but allows $\mathrm{Var}(Y_i\mid d_i)={\mu }_i+{\mu }_i^2/\theta$. It improves the residual deviance from $75.806$ to $18.011$, close to its $16$ residual degrees of freedom, and lowers the Akaike information criterion from $172.34$ to $141.66$. Both comparisons strongly favour the negative-binomial fit, although its dose coefficient remains statistically insignificant.

The variance $\mu +{\mu }^2/\theta$ approaches the Poisson distribution variance $\mu$ when $\theta \to \infty$. Equivalently, with $\alpha =1/\theta \ge 0$, the Poisson model is the boundary value $\alpha =0$. The usual Wilks theorem for a likelihood-ratio test assumes that the null parameter is an interior point of a smooth parameter space, so comparing the statistic with an ordinary ${\chi }_1^2$ law is invalid here.

A valid parametric bootstrap proceeds as follows. First fit the null Poisson model and retain its fitted means ${\hat{\mu }}_i$. For each bootstrap repetition $b=1,\dots ,B$, independently simulateusing the original doses, refit both the Poisson and negative-binomial models to that simulated data, and calculateFor the observations calculate the analogous $T_{obs}$. The bootstrap $p$-valueuses the null distribution with its boundary and finite-sample fitting behaviour automatically reproduced. A large $B$ controls the Monte Carlo error of this estimate.

Because the columns of the design matrix are centred, ridge regression with an unpenalized intercept solvesThe normal equations give, for $\lambda >0$,The fitted values are consequentlyIf the objective is normalized by $n$, the same formulas hold after replacing $\lambda$ by $n\lambda$.

Choose $\lambda$ to minimize an estimate of out-of-sample mean squared prediction error, commonly K-fold cross-validation or a separate validation set. As $\lambda$ increases, the coefficient vector is shrunk toward zero. This generally increases bias of an estimator but decreases variance of an estimator; the minimizing value balances the two contributions in the bias-variance tradeoff. The independent test set in the question can assess the final choice, but repeatedly selecting $\lambda$ on that same set would cause data leakage.

Coordinate descent cycles through the intercept and coefficient coordinates, minimizing the convex ridge objective in one coordinate while holding the others fixed. Given current coefficients, updateFor coordinate $j$, form the partial residualand update it by the exact one-dimensional minimizerRepeated sweeps converge to the unique fitted value because the objective is a convex function; with $\lambda >0$ it is strictly convex in $\beta$.

The solid curve is the test errorand the dashed curve is the training error $n^{-1}\Vert Y-{\hat{Y}}_{\lambda }{\Vert }_2^2$. As the horizontal coordinate tends to $+\infty$, $\lambda \to \infty$, so every penalized slope tends to zero and ${\hat{Y}}_{\lambda }\to \overline{Y}1$. The two limits are thereforerespectively.

As the horizontal coordinate tends to $-\infty$, $\lambda \downarrow 0$ and the fit approaches the ordinary least squares fit $H_{0}Y$. Hence the solid curve tends to $n^{-1}\Vert Y^{\ast }-H_{0}Y{\Vert }_2^2$, which the plot shows is approximately $0.4$.

Put $f=(f(x_1),\dots ,f(x_n){)}^T$, so $Y=f+\epsilon$ and $Y^{\ast }=f+{\epsilon }^{\ast }$ with independent noise vectors having covariance matrix $I_n$. For any deterministic linear smoother $H$,whereas independence givesExpanding the first trace shows that the second expression exceeds the first by $2\mathrm{tr}(H)$, proving the identity.

For ridge regression,Its effective degrees of freedom are $\mathrm{tr}(H_{\lambda })$. Thus training error is optimistically biased for independent-copy prediction error by $2\mathrm{tr}(H_{\lambda })/n$. The graph exhibits exactly this effect: the dashed training curve keeps falling as $\lambda$ decreases, while the solid test curve eventually rises through overfitting.

The classification form of CART starts with the root rectangle $R=[0,1{]}^2$. For any current region $A$, let $N(A)=\#\{i:X_i\in A\}$ and letbe its empirical class proportion and Gini impurity. A candidate axis-aligned split $X_j\le s$ partitions $R$ into $U$ and $V$. Its impurity change isAmong all coordinates and thresholds between consecutive observed coordinates, CART chooses a split minimizing $Q$, then applies the same recursive partitioning independently to the children until a stopping rule is met. Each terminal region predicts its majority class. Pruning may then select a smaller subtree by penalizing the number of leaves.

For the resulting classifier $\hat{C}$, the training error iswhile its prediction error is $R(\hat{C})=\mathbb{P}\{\hat{C}(X_{new})\ne Y_{new}\}$ for an independent observation drawn from the target population.

Let $w=N(U)/N(R)$, so $1-w=N(V)/N(R)$. Since class counts add,The function $g(p)=p(1-p)$ is concave function on $[0,1]$. Jensen inequality therefore giveswhich is precisely $Q\le 0$. Thus an axis-aligned split cannot increase the weighted empirical Gini impurity.

A random forest fits each of its $500$ classification trees to an independent bootstrap sample of the nine observations. At each node it draws mtry=2 candidate coordinates; because the data have exactly two coordinates, both are available, and a CART impurity calculation chooses the split. The trees are grown deeply without ordinary cost-complexity pruning, and their majority vote is the forest prediction.

R reports an out-of-bag error estimate: an observation is predicted only by trees whose bootstrap samples omitted it. The confusion matrix says that class 1 has three correct and three incorrect out-of-bag predictions, while class 2 has one correct and two incorrect predictions. Hence the total out-of-bag error is $5/9$.

The loop performs Leave-one-out cross-validation: for each $i$ it fits a forest to the other eight observations, tests it on observation $i$, and averages the nine zero-one losses. A random forest's out-of-bag error estimate approximates the same held-out prediction error from one fit, because each tree automatically omits roughly a proportion $e^{-1}$ of the observations in its bootstrap sample.

The one-nearest-neighbour classification boundary consists of the portions of the Voronoi diagram separating cells whose observed labels differ. For these nine grid points it forms diagonal and vertical or horizontal perpendicular-bisector segments around the three triangular observations. Every training point is its own nearest neighbour, so, absent a distance tie convention that excludes the query itself, its training error is zero.

An unpruned maximal CART classifier repeatedly cuts with vertical or horizontal lines until every terminal rectangle is pure or contains observations that cannot be separated by an axis-aligned split. Here the distinct grid points can be isolated into pure rectangles, producing a step-shaped, axis-aligned decision boundary and zero training error.

The random forest boundary is the majority vote of many bootstrap-grown axis-aligned trees. It remains piecewise axis-aligned but averages away many unstable individual cuts, so one should sketch a less extreme boundary enclosing regions supported repeatedly by the triangular points. Its resubstitution training error is typically small and can be zero, but bootstrap omission and voting mean that zero is not guaranteed. The relevant built-in estimate is instead the out-of-bag error, which the output gives as $5/9$; this large value reflects the tiny sample and unstable labels near the class boundary.

For $X_i^{\ast }=(1,X_i^T{)}^T$ and $\gamma =({\gamma }_0,{\beta }^T{)}^T$, a linear support vector machine predictsOne penalized formulation minimizes empirical hinge loss plus a squared Euclidean norm penalty:Conventions often leave the intercept unpenalized, replacing $\Vert \gamma {\Vert }_2^2$ by $\Vert \beta {\Vert }_2^2$; this does not change the role of the two terms.

A separating hyperplane for signed data satisfies $Y_i({\gamma }_0+X_i^T\beta )>0$ for every $i$; its geometric set is $\{x:{\gamma }_0+x^T\beta =0\}$. The plot marks three support vectors. At the shown fit they lie on the two support-vector-machine margin boundaries, so their signed functional margins satisfyThe solid line is the decision hyperplane $X^{\ast T}\hat{\gamma }=0$, while the dashed parallel lines are $X^{\ast T}\hat{\gamma }=1$ and $X^{\ast T}\hat{\gamma }=-1$.

If $\lambda =0$, every parameter vector with all margins at least one has zero hinge loss. Scaling or changing a separating vector can therefore give another minimizer, so the objective need not select the displayed maximum-margin direction or the same three lines. Positive quadratic regularization selects a finite, minimum-norm compromise.

This is a normalized perceptron algorithm. Because $\Vert Z_i{\Vert }_2=1$, an update on a misclassified point obeysHere the two inequalities use the update condition $Y_i{Z}_i^T{\gamma }^{(m)}\le 0$ and the assumed unit margin $Y_i{Z}_i^T\hat{\gamma }\ge 1$. A squared distance cannot become negative, so there can be at most $\Vert {\gamma }^{(0)}-\hat{\gamma }{\Vert }_2^2$ updates. The algorithm then returns a vector that correctly separates every training point. The estimate $\hat{\gamma }$ is the comparison vector for the proof; without an additional uniqueness condition the returned separator need not equal that particular vector.

A binary logistic regression setsand classifies by the sign of ${\gamma }_0+x^T\beta$. Its unpenalized maximum-likelihood estimator minimizes the empirical logistic lossThe plotted data are complete separation data: there is a vector $v$ with every signed margin $Y_i{X}_i^{\ast T}v>0$. For every finite $c>0$, increasing $c$ strictly decreases each term of $L(cv)$, and $L(cv)\to 0$ as $c\to \infty$. No finite parameter attains zero, so the unpenalized optimization has no solution.

Adding an $L^2$ penalty $\lambda \Vert \gamma {\Vert }_2^2$ with $\lambda >0$, constraining $\Vert \gamma {\Vert }_2$, or using a finite stopping rule makes the problem attain a finite approximate solution. The penalized option is preferable because cross-validation can select the strength of regularization.