Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 218 / 4

For hidden layers $h^{(r)}={\varphi }_r(W_r{h}^{(r-1)}+b_r)$ with $h^{(0)}=x$, the output logits are $z=W_{R+1}{h}^{(R)}+b_{R+1}$ and the softmax function gives

Multinomial logistic regression is the network with no hidden layer: $z=Wx+b$ followed by softmax. Binary logistic classification uses one logit $w^{T}x+b$ followed by the sigmoid function.

Deep sigmoid networks suffer from vanishing gradients because derivatives become small in saturated units and multiply across layers. The ReLU has derivative one on its positive half-line and therefore propagates gradients more effectively there.

For all weights and biases collected in $\vartheta$, fitUse backpropagation with stochastic gradient descent or a modern adaptive variant, treating the absolute-value derivative at zero as stated. Select $({\lambda }_1,{\lambda }_2)$ by validation or cross-validation and refit using the selected pair.

A dead ReLU unit has nonpositive preactivation for every training input, so its output and gradient are always zero. Replace $\max (0,z)$ by the leaky ReLUIts negative-side derivative $a$ permits recovery while it remains nonsaturating, preserving the gradient advantage over sigmoid activation.