= Neural tangent kernel
{title2=$k_{\mathrm{NTK}}(x,x')$}
For a differentiable <neural network> $f(\theta,x)$ and initialization law $P$, the expected neural tangent kernel is $k_{\mathrm{NTK}}(x,x')=\mathbb E_{\theta\sim P}\langle\nabla_\theta f(\theta,x),\nabla_\theta f(\theta,x')\rangle$, assuming finite diagonal <expectations>. At a fixed parameter value the corresponding <inner product> defines the empirical neural tangent kernel. Both are <positive semidefinite kernels>, since a finite <quadratic form> is respectively an expected squared <norm> or a squared <norm>. In suitable infinite-width scalings, the empirical <kernel matrix> converges to a deterministic limit and remains constant during training. Squared-error <gradient flow> then has the same training-prediction dynamics as <kernel regression> with this limit, provided the initial predictions match.
Back to article page
