= Hessian of a sum of functions of linear forms
{title2=$\nabla^2l(x)=A^{\mathsf T}\operatorname{diag}(y_i f''(a_i^{\mathsf T}x))A$}
For twice differentiable $f:\mathbb R\to\mathbb R$, constant <vectors> $a_i$ and constant scalars $y_i$, let $l(x)=\sum_i y_i f(a_i^{\mathsf T}x)$, and let row $i$ of $A$ be $a_i^{\mathsf T}$. The <chain rule> gives
$$
\nabla l(x)=\sum_i y_i f'(a_i^{\mathsf T}x)a_i,\qquad
\nabla^2l(x)=\sum_i y_i f''(a_i^{\mathsf T}x)a_i a_i^{\mathsf T}.
$$
The second formula is the displayed <matrix> product because entry $(j,k)$ is the same weighted sum $\sum_i y_i f''(a_i^{\mathsf T}x)a_{ij}a_{ik}$. Signed diagonal factors do not ensure a <positive-definite> <Hessian matrix>. The factorization also gives an efficient <matrix-free Hessian-vector product>.
Back to article page
