Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 2 / c

Let $J(t)$ be the $n\times d$ Jacobian matrix with row $J_i(t)={\nabla }_{\theta }f(\theta (t),x_i{)}^T$. Differentiating the squared-error loss function givesThe parameter gradient flow is consequently $\dot{\theta }=J^T(y-u)$. The chain rule then giveswhere, explicitly,It is a Gram matrix and therefore a positive semidefinite matrix: $v^{T}H(t)v=\Vert J(t{)}^{T}v{\Vert }_2^2\ge 0$. These derivatives are understood wherever the network is differentiable; for piecewise differentiable activations the same calculation applies along differentiable portions of the trajectory.

Under the stated infinite-width scaling and freezing assumptions, the deterministic limiting kernel matrix has entriesFor a sequence of finite-width networks, this expectation is interpreted in the stated width limit. The resulting neural tangent kernel is a positive-semidefinite kernel, since every finite coefficient vector $a$ gives $a^T{H}^{\ast }a=\mathbb{E}\Vert {\sum }_i{a}_i{\nabla }_{\theta }f(\theta (0),x_i){\Vert }_2^2\ge 0$.

The prediction gradient flow is linear:The matrix exponential solves the constant-coefficient system because the residual $u-y$ satisfies $\frac{d}{dt}(u-y)=-H^{\ast }(u-y)$. Each positive-eigenvalue component decays exponentially; a component in the null space of $H^{\ast }$ stays at its initial residual. Thus $u(t)\to y$ for every initial prediction if $H^{\ast }$ is a positive-definite matrix. The limiting kernel matrix being deterministic does not require $u(0)$ to be deterministic.

For $h$ in the Reproducing kernel Hilbert space ${\mathcal{H}}_k$, take the directional derivative:The reproducing property says $h(x_i)=⟨k(x_i,\cdot ),h{⟩}_{{\mathcal{H}}_k}$. HenceEach kernel section belongs to the Reproducing kernel Hilbert space, so this finite sum is an admissible element. Uniqueness of a vector with these inner products gives the functional gradient

Evaluating the negative functional gradient at $x_j$ yieldsThus the evaluation vector $v(t)=(g_t(x_1),\dots ,g_t(x_n){)}^T$ followsThis is exactly the limiting prediction dynamics in part (ii). If $v(0)=u(0)$, subtracting the two systems gives $\frac{d}{dt}(v-u)=-H^{\ast }(v-u)$ with zero initial value, so $v(t)=u(t)$ for every $t$.

An infinitely wide neural network in this regime follows the same training-prediction dynamics as kernel regression with its neural tangent kernel and matching initial predictions. For this squared-error loss function, the relevant algorithm is gradient flow in the Reproducing kernel Hilbert space. Matching predictions at arbitrary additional inputs also requires the corresponding cross-input neural tangent kernel to freeze and the initial predictions there to match. A finite-width neural network can change its feature map during training, so the equivalence depends on the assumptions rather than holding for every architecture and parameterization.