Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 2 / c / iv / Solution

Evaluating the negative functional gradient at $x_j$ yieldsThus the evaluation vector $v(t)=(g_t(x_1),\dots ,g_t(x_n){)}^T$ followsThis is exactly the limiting prediction dynamics in part (ii). If $v(0)=u(0)$, subtracting the two systems gives $\frac{d}{dt}(v-u)=-H^{\ast }(v-u)$ with zero initial value, so $v(t)=u(t)$ for every $t$.

An infinitely wide neural network in this regime follows the same training-prediction dynamics as kernel regression with its neural tangent kernel and matching initial predictions. For this squared-error loss function, the relevant algorithm is gradient flow in the Reproducing kernel Hilbert space. Matching predictions at arbitrary additional inputs also requires the corresponding cross-input neural tangent kernel to freeze and the initial predictions there to match. A finite-width neural network can change its feature map during training, so the equivalence depends on the assumptions rather than holding for every architecture and parameterization.