 Solution

Under the stated infinite-width scaling and freezing assumptions, the deterministic limiting kernel matrix has entriesFor a sequence of finite-width networks, this expectation is interpreted in the stated width limit. The resulting neural tangent kernel is a positive-semidefinite kernel, since every finite coefficient vector $a$ gives $a^T{H}^{\ast }a=\mathbb{E}\Vert {\sum }_i{a}_i{\nabla }_{\theta }f(\theta (0),x_i){\Vert }_2^2\ge 0$.

The prediction gradient flow is linear:The matrix exponential solves the constant-coefficient system because the residual $u-y$ satisfies $\frac{d}{dt}(u-y)=-H^{\ast }(u-y)$. Each positive-eigenvalue component decays exponentially; a component in the null space of $H^{\ast }$ stays at its initial residual. Thus $u(t)\to y$ for every initial prediction if $H^{\ast }$ is a positive-definite matrix. The limiting kernel matrix being deterministic does not require $u(0)$ to be deterministic.