= Solution
Under the stated infinite-width scaling and freezing assumptions, the deterministic limiting <kernel matrix> has entries
$$
\boxed{H^*_{ij}=\mathbb E_{\theta(0)\sim P}\left\langle\nabla_\theta f(\theta(0),x_i),\nabla_\theta f(\theta(0),x_j)\right\rangle=k_{\rm NTK}(x_i,x_j)}.
$$
For a sequence of finite-width networks, this expectation is interpreted in the stated width limit. The resulting <neural tangent kernel> is a <positive-semidefinite kernel>, since every finite coefficient <vector> $a$ gives $a^TH^*a=\mathbb E\|\sum_i a_i\nabla_\theta f(\theta(0),x_i)\|_2^2\geq0$.
The prediction <gradient flow> is linear:
$$
\boxed{\dot u=H^*(y-u),\qquad u(t)=y+e^{-tH^*}(u(0)-y)}.
$$
The <matrix exponential> solves the constant-coefficient system because the residual $u-y$ satisfies $\frac d{dt}(u-y)=-H^*(u-y)$. Each positive-<eigenvalue> component decays exponentially; a component in the <null space> of $H^*$ stays at its initial residual. Thus $u(t)\to y$ for every initial prediction if $H^*$ is a <positive-definite matrix>. The limiting <kernel matrix> being deterministic does not require $u(0)$ to be deterministic.
Back to article page
