Past exam of the mathematics course of the University of Cambridge / 2026 / iii / Paper 218 / 5

For standardized $x\in {\mathbb{R}}^8$, model1 computesfollowed by logits $z=W_3{h}_2+b_3\in {\mathbb{R}}^2$ and softmax function probabilitiesThe parameter count isFor one-hot labels $y_{ik}$, the categorical cross-entropy loss isThis is the negative conditional log-likelihood of independent categorical labels, equivalently Bernoulli labels in the two-class case.

Solved by gpt-5.6-sol high.

Stochastic gradient descent replaces the full empirical-loss gradient by the gradient on a randomly ordered observation or mini-batch, then updates $\theta \leftarrow \theta -\eta \hat{\nabla }L(\theta )$. The training half contains $768/2=384$ observations, so batches of 16 give $384/16=24$ updates per epoch. Over 100 epochs every parameter is updated $2400$ times.

Solved by gpt-5.6-sol high.

The neural-network likelihood is nonconvex, so optimization can stop at a local optimum or saddle rather than a global maximum. One hundred epochs may be insufficient for convergence. Mini-batch gradient noise together with a fixed positive learning rate can keep the iterates fluctuating around a stationary point rather than reaching it exactly. Any of these prevents the final parameters from being exact maximum-likelihood estimators.

Solved by gpt-5.6-sol high.

The dashed curve is training accuracy: it continues to rise as optimization adapts to the training observations. The solid curve is testing accuracy: it peaks near 10 epochs and then declines. The widening gap is overfitting.

A sensible choice is about 10 epochs, selected by early stopping at the maximum validation accuracy. In a proper analysis, a validation set rather than the final test set should choose this epoch. Early stopping is an implicit regularization method because it limits how far the parameters can adapt to training-specific noise.

Solved by gpt-5.6-sol high.