Past exam of the mathematics course of the University of Cambridge / 2025 / iii / Paper 218 / 4

Encode the two classes by the standard basis vectors of ${\mathbb{R}}^2$. With rectified linear unit $r(t)=\max (t,0)$ applied coordinatewise and softmax function $s_j(z)=e^{z_j}/{\sum }_k{e}^{z_k}$, the fitted feedforward neural network iswhere $W_1\in {\mathbb{R}}^{40\times 40}$, $b_1\in {\mathbb{R}}^{40}$, $W_2\in {\mathbb{R}}^{20\times 40}$, $b_2\in {\mathbb{R}}^{20}$, $W_3\in {\mathbb{R}}^{2\times 20}$, and $b_3\in {\mathbb{R}}^2$. The number of trainable parameters is

Training minimizes the empirical categorical cross-entropy lossby stochastic gradient descent. The independent validation set monitors generalization, while the small fixed number of epochs limits how long the network can fit training noise; using validation loss for early stopping would make this safeguard explicit. A forward pass computes all layer activations, class probabilities, and the mini-batch loss. There are $10000/2=5000$ training mini-batches per epoch and hence $5\cdot 5000=25000$ training forward passes, in addition to validation evaluation after each epoch.

The usual Akaike information criterion correction assumes a regular maximum-likelihood fit with a meaningful fixed parameter dimension. Here stochastic optimization stopped after five epochs need not attain the maximum likelihood estimator, and neural-network symmetries, inactive units, and heavy overparameterization make the raw count $2502$ a poor effective dimension. Either failure invalidates a direct AIC comparison with an ordinary logistic regression.

The proposed function $g(\eta )=\eta \Phi (\eta )$ is the Gaussian error linear unit. Unlike ReLU, it is smooth at zero and has a nonzero gradient on much of the negative half-line, reducing dead hidden units and making gradient optimization smoother. It is less computationally convenient because evaluating the standard normal cumulative distribution function is costlier than taking a maximum, and it does not produce ReLU's exact sparse zero activations.

The intended quantity is the Leave-one-out cross-validation errorwhere ${\hat{h}}^{(-i)}$ is trained without observation $i$ and the loss is the zero-one misclassification loss. Computing it literally requires fitting $10000$ neural networks, which is prohibitively expensive.

The assignment nn.model.i <- nn.model does not construct a fresh untrained Keras model: it aliases an object whose weights were already fitted using every training observation, including the nominally held-out one, and repeated fits continue mutating those weights. This data leakage makes metric2 severely optimistic. Moreover, random leave-one-out validation among reviews from 2012--2025 does not reproduce the dataset shift to new recent reviews that metric1 measures.