Define the <softmax function>[softmax weights]
$$
p_i(x)=\frac{e^{\beta z_i(x)}}{\sum_j e^{\beta z_j(x)}}.
$$
They obey $p_i\geq0$ and $\sum_i p_i=1$. The <gradient> is their weighted mean,
$$
\boxed{\nabla f_\beta(x)=\sum_i p_i(x)a_i.}
$$
Differentiating once more gives the covariance-form <Hessian matrix>
$$
\nabla^2f_\beta(x)
=\beta\left(\sum_i p_i a_i a_i^T-\bar a\bar a^T\right),
\qquad \bar a=\sum_i p_i a_i.
$$
For every unit <vector> $u$,
$$
u^T\nabla^2f_\beta(x)u
=\beta\operatorname{Var}_{i\sim p}(u^Ta_i)
\leq\beta\sum_i p_i(u^Ta_i)^2
\leq\beta G^2,
$$
where $G=\max_i\|a_i\|_2$. The Hessian is a <covariance matrix>, so it is <positive semidefinite matrix>[positive semidefinite]; the displayed upper bound also gives $\nabla^2f_\beta\preceq\beta G^2I$ in the <Loewner order>. Consequently $f_\beta$ has a <Lipschitz gradient> with
$$
\boxed{L=\beta G^2.}
$$

Solution

Past exam of the mathematics course of the University of Cambridge / 2023 / iii / Paper 339 / 1 / d / Solution

 Solution

ID: past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2023/iii/paper-339/1/d/solution