Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 2

In this setting a positive-semidefinite kernel is an inner-product similarity: there exist a real Hilbert space $\mathcal{H}$ and a feature map $\phi :\mathcal{X}\to \mathcal{H}$ such thatThe feature map may be infinite-dimensional. Equivalently, $k$ is symmetric and every finite kernel matrix is a positive semidefinite matrix. The equivalence follows by forming the canonical feature space of a positive-semidefinite kernel and taking its Hilbert space completion.

For every $m\ge 1$, every choice $x_1,\dots ,x_m\in \mathcal{X}$, and every coefficient vector $\alpha \in {\mathbb{R}}^m$, the required condition isThus ${\alpha }^{T}K\alpha \ge 0$ for the kernel matrix $K$. For a feature map, this follows fromIn kernel methods, the term positive-definite kernel commonly uses this non-strict inequality. Strict positive definiteness is the stronger requirement that the expression be positive for distinct inputs and every nonzero $\alpha$; it is not required here.

The closure property of positive-semidefinite kernels gives three constructions, each seen directly from the feature maps ${\phi }_1,{\phi }_2$.

Another useful consequence is pullback: if $h:\mathcal{X}\to \mathcal{Z}$ and $k_0$ is a positive-semidefinite kernel on $\mathcal{Z}$, then $k_0(h(x),h(x^{\prime }))$ has feature map ${\phi }_0\circ h$.

Take the real L2 space $\mathcal{H}=L^2((0,\infty ),dt)$ and the feature mapEach indicator function belongs to $\mathcal{H}$ because its squared L2 norm is $x<\infty$. Their inner product is the length of the overlapping intervals:This realizes $k_1$ as a positive-semidefinite kernel; it is also the Brownian covariance kernel.

Choose two inputs $0<a<b$. Their proposed kernel matrix isThe coefficient vector $\alpha =(1,-1{)}^T$ givesThus $K$ is not a positive semidefinite matrix, andEquivalently, its determinant $ab-b^2=b(a-b)$ is negative.

Reciprocation reverses the order of positive numbers, soThis is the pullback of the positive-semidefinite kernel from part (i) by $x\mapsto 1/x$. Explicitly, the feature map into the same L2 space isHence $k_3$ is a positive-semidefinite kernel, with $\Vert \psi (x){\Vert }_2^2=1/x<\infty$ for every allowed input.

Let $J(t)$ be the $n\times d$ Jacobian matrix with row $J_i(t)={\nabla }_{\theta }f(\theta (t),x_i{)}^T$. Differentiating the squared-error loss function givesThe parameter gradient flow is consequently $\dot{\theta }=J^T(y-u)$. The chain rule then giveswhere, explicitly,It is a Gram matrix and therefore a positive semidefinite matrix: $v^{T}H(t)v=\Vert J(t{)}^{T}v{\Vert }_2^2\ge 0$. These derivatives are understood wherever the network is differentiable; for piecewise differentiable activations the same calculation applies along differentiable portions of the trajectory.

Under the stated infinite-width scaling and freezing assumptions, the deterministic limiting kernel matrix has entriesFor a sequence of finite-width networks, this expectation is interpreted in the stated width limit. The resulting neural tangent kernel is a positive-semidefinite kernel, since every finite coefficient vector $a$ gives $a^T{H}^{\ast }a=\mathbb{E}\Vert {\sum }_i{a}_i{\nabla }_{\theta }f(\theta (0),x_i){\Vert }_2^2\ge 0$.

The prediction gradient flow is linear:The matrix exponential solves the constant-coefficient system because the residual $u-y$ satisfies $\frac{d}{dt}(u-y)=-H^{\ast }(u-y)$. Each positive-eigenvalue component decays exponentially; a component in the null space of $H^{\ast }$ stays at its initial residual. Thus $u(t)\to y$ for every initial prediction if $H^{\ast }$ is a positive-definite matrix. The limiting kernel matrix being deterministic does not require $u(0)$ to be deterministic.

For $h$ in the Reproducing kernel Hilbert space ${\mathcal{H}}_k$, take the directional derivative:The reproducing property says $h(x_i)=⟨k(x_i,\cdot ),h{⟩}_{{\mathcal{H}}_k}$. HenceEach kernel section belongs to the Reproducing kernel Hilbert space, so this finite sum is an admissible element. Uniqueness of a vector with these inner products gives the functional gradient

Evaluating the negative functional gradient at $x_j$ yieldsThus the evaluation vector $v(t)=(g_t(x_1),\dots ,g_t(x_n){)}^T$ followsThis is exactly the limiting prediction dynamics in part (ii). If $v(0)=u(0)$, subtracting the two systems gives $\frac{d}{dt}(v-u)=-H^{\ast }(v-u)$ with zero initial value, so $v(t)=u(t)$ for every $t$.

An infinitely wide neural network in this regime follows the same training-prediction dynamics as kernel regression with its neural tangent kernel and matching initial predictions. For this squared-error loss function, the relevant algorithm is gradient flow in the Reproducing kernel Hilbert space. Matching predictions at arbitrary additional inputs also requires the corresponding cross-input neural tangent kernel to freeze and the initial predictions there to match. A finite-width neural network can change its feature map during training, so the equivalence depends on the assumptions rather than holding for every architecture and parameterization.