= Solution
Let $J(t)$ be the $n\times d$ <Jacobian matrix> with row $J_i(t)=\nabla_\theta f(\theta(t),x_i)^T$. Differentiating the squared-error <loss function> gives
$$
\nabla_\theta\ell(\theta(t))=\sum_i(u_i(t)-y_i)\nabla_\theta f(\theta(t),x_i)=J(t)^T(u(t)-y).
$$
The parameter <gradient flow> is consequently $\dot\theta=J^T(y-u)$. The <chain rule> then gives
$$
\boxed{\dot u(t)=H(t)(y-u(t)),\qquad H(t)=J(t)J(t)^T},
$$
where, explicitly,
$$
\boxed{H_{ij}(t)=\left\langle\nabla_\theta f(\theta(t),x_i),\nabla_\theta f(\theta(t),x_j)\right\rangle_{\mathbb R^d}}.
$$
It is a <Gram matrix> and therefore a <positive semidefinite matrix>: $v^TH(t)v=\|J(t)^Tv\|_2^2\geq0$. These derivatives are understood wherever the network is differentiable; for piecewise differentiable activations the same calculation applies along differentiable portions of the trajectory.
Back to article page
