Past exam of the chemistry course of the year 4 of the natural sciences course of the University of Cambridge / 2022 / Paper 3 / Section F / 47

Let $z_{ij}$ be the input to output weight $w_j$ for datum $i$, including $z_{i0}=1$ for the bias. With identity output activation, $h_i=a_i={\sum }_j{w}_j{z}_{ij}$. The chain rule givesThe $z_{ij}$ are held fixed when differentiating an output-layer weight; they may be hidden-layer activations rather than raw descriptors. Hidden-weight derivatives require backpropagation through the preceding layers.

The printed PDF has a plus sign before the second logarithm. Differentiating that expression literally, with $h_i=\sigma (a_i)$ and $\partial h_i/\partial w_j=h_i(1-h_i)z_{ij}$, givesFor example, at $y_i=0$ it is $-h_i{z}_{ij}$, so minimizing this printed objective pushes the predicted probability of class one upwards rather than downwards; the objective is unbounded below as $h_i\to 1$. It is therefore not the usual binary cross-entropy.

If the second sign was intended to be negative, the standard negative log-likelihood is $-y_i\log h_i-(1-y_i)\log (1-h_i)$ and the derivative simplifies instead to $(h_i-y_i)z_{ij}$. Stating both cases preserves the actual question's algebra while identifying the apparent sign error. The logarithms here are natural; another base multiplies the gradients by its constant conversion factor.

For $K$ mutually exclusive classes, the output layer should provide $K$ nonnegative probabilities summing to one, with labels aligned consistently to output coordinates. A softmax activation on logits $a_k$ providesSubtracting ${\max }_{\ell }{a}_{\ell }$ from all logits before exponentiation preserves these probabilities and prevents numerical overflow. Categorical cross-entropy $-{\sum }_k{y}_k\log p_k$ is appropriate for one-hot or probabilistic class targets. Independent sigmoid outputs fit a different setting in which multiple labels can be present simultaneously and their probabilities need not sum to one.

Compose affine maps with nonlinear hidden activation functions, for example $h(x)=W_2\mathrm{ReLU}(W_{1}x+b_1)+b_2$. Several hidden units can respond to different linear combinations of input features; later units combine their responses to represent curved or piecewise-linear decision boundaries and interactions. A product interaction can be understood from $xy=[(x+y{)}^2-(x-y{)}^2]/4$: units approximating nonlinear functions of $x+y$ and $x-y$ can represent effects that no single additive linear fit captures.

Nonlinearity in hidden layers is essential: with only identity activations, repeated affine maps collapse to one affine map regardless of depth. Width and depth determine available representations; regularization, validation and suitable optimization are needed to learn useful rather than overfitted interactions. A nonlinear output activation alone applied to one linear score does not give an unrestricted model of feature interactions.

A multilayer perceptron usually connects every input unit of a layer to every output unit. For $n$ inputs and $m$ outputs its dense affine layer has $nm+m$ parameters, including biases. A convolutional layer instead applies small local filters with weight sharing across positions. A two-dimensional filter of size $k_h\times k_w$, with $C_{\mathrm{in}}$ input channels and $C_{\mathrm{out}}$ output channels, has $k_h{k}_w{C}_{\mathrm{in}}{C}_{\mathrm{out}}+C_{\mathrm{out}}$ parameters. The count does not increase with spatial height and width, although the activation storage and arithmetic do.

Local receptive fields and repeated layers capture increasingly extended patterns. translation equivariance follows from shared filters under compatible boundary conventions; pooling or global aggregation can help build invariance but a convolution alone is not generally translation-invariant. Flattening a large convolutional output into a dense head can reintroduce many parameters. For an unordered descriptor vector, a convolutional locality assumption must be justified rather than imposed merely because convolutional layers have fewer weights.