Past exam of the mathematics course of the University of Cambridge / 2023 / iii / Paper 218 / 4

With $f=1_{\{\mathrm{font}=\mathrm{serif}\}}$ and $d=1_{\{\mathrm{display}=\mathrm{popup}\}}$, the model-matrix input isThe feedforward neural network has three inputs, a fully connected layer of two ReLU units, and a fully connected two-class softmax output. Algebraically,where $W\in {\mathbb{R}}^{2\times 3}$, $b\in {\mathbb{R}}^2$, $V\in {\mathbb{R}}^{2\times 2}$, and $c\in {\mathbb{R}}^2$. The coding is $0$ for no click and $1$ for a click. There areparameters.

The fitted classifier isand training minimizes the empirical categorical cross-entropy loss

The softmax non-identifiability already proves that the coefficient vector is not unique. For any $a\in {\mathbb{R}}^2$ and $r\in \mathbb{R}$, replace both rows of $V$ by $V_{k\cdot }+a^T$ and both output biases by $c_k+r$. Every logit then gains the same value $a^{T}h+r$, so all softmax probabilities, classifications, and losses remain unchanged. Permuting the two hidden units supplies another non-uniqueness.

For the one-layer softmax fit, write its class logits as $z_k=a_k+b_k^{T}x$. ThenThus an equivalent logistic regression classifier predicts a click exactly whenUsing one reference class removes the common-logit-shift non-identifiability. Subject to the usual full-rank and no-separation conditions, the logistic parameter is identifiable. It uses four effective parameters rather than an eight-parameter redundant softmax representation, has a convex loss, and gives directly interpretable log-odds coefficients, so it is preferable for this binary linear classifier.

The first observation has $x=(1,0,0{)}^T$ and one-hot label $y=(0,1{)}^T$ in the output order $(\mathrm{no},\mathrm{yes})$. If every kernel weight and bias initially equals one, each hidden preactivation is two, so $h=(2,2{)}^T$. Both logits equal five and $p=(1/2,1/2{)}^T$.

For stochastic gradient descent on one cross-entropy observation,Hence the output-weight gradient has first row $(1,1)$ and second row $(-1,-1)$, while the output-bias gradient is $(1/2,-1/2)$. With learning rate one,Using the old output weights for backpropagation givesso every entry of $W$ and $b$ remains equal to one after this batch.