Past exam of the mathematics course of the University of Cambridge / 2019 / iii / Paper 218 / 5

With column vectors and learned biases,The dimensions are $16\to 100\to 75\to 50\to 26$. The number of parameters is

The loop divides each feature by its sample maximum, putting features with nonnegative values on comparable scales and improving numerical conditioning for gradient optimization. If exactly one hidden layer must use the sigmoid function, use layer 3. Sigmoid derivatives can become small in saturated regions; placing it latest minimizes the number of subsequent gradient multiplications affected by this saturation, while the earlier ReLU layers retain efficient gradient propagation.

For one-hot labels $y_{il}$ and predicted probabilities $p_{il}(\theta )$, the categorical cross-entropy loss isStochastic gradient descent initializes $\theta$, randomly orders the observations in each of five epochs, and for each single-observation batch computes a forward pass, the sample loss, and its gradient, then updatesBackpropagation is used after the forward loss evaluation to compute this gradient from the output layer back through the hidden layers.

With identity hidden activations and no biases, the pre-softmax map is a product of weight matrices. For any nonzero scalar $c$, multiplying the incoming weights of one hidden neuron by $c$ and dividing its outgoing weights by $c$ leaves that product, every output probability, and the cross-entropy unchanged. Every minimizer therefore belongs to an infinite continuum of equivalent parameterizations; more generally, invertible changes of hidden coordinates and their inverse in the adjacent layer give the same network function.

I would use batch_size=1. The resulting stochastic-gradient noise helps move along flat nonidentifiable directions and escape saddle regions, whereas full-batch gradient descent is deterministic and can stagnate in this highly nonconvex, singular parameterization.