Past exam of the mathematics course of the University of Cambridge / 2023 / iii / Paper 218 / 4 / d / Solution

The first observation has $x=(1,0,0{)}^T$ and one-hot label $y=(0,1{)}^T$ in the output order $(\mathrm{no},\mathrm{yes})$. If every kernel weight and bias initially equals one, each hidden preactivation is two, so $h=(2,2{)}^T$. Both logits equal five and $p=(1/2,1/2{)}^T$.

For stochastic gradient descent on one cross-entropy observation,Hence the output-weight gradient has first row $(1,1)$ and second row $(-1,-1)$, while the output-bias gradient is $(1/2,-1/2)$. With learning rate one,Using the old output weights for backpropagation givesso every entry of $W$ and $b$ remains equal to one after this batch.