 Neural tangent kernel

For a differentiable neural network $f(\theta ,x)$ and initialization law $P$, the expected neural tangent kernel is $k_{\mathrm{NTK}}(x,x^{\prime })={\mathbb{E}}_{\theta \sim P}⟨{\nabla }_{\theta }f(\theta ,x),{\nabla }_{\theta }f(\theta ,x^{\prime })⟩$, assuming finite diagonal expectations. At a fixed parameter value the corresponding inner product defines the empirical neural tangent kernel. Both are positive-semidefinite kernels, since a finite quadratic form is respectively an expected squared norm or a squared norm. In suitable infinite-width scalings, the empirical kernel matrix converges to a deterministic limit and remains constant during training. Squared-error gradient flow then has the same training-prediction dynamics as kernel regression with this limit, provided the initial predictions match.