Past exam of the mathematics course of the University of Cambridge / 2023 / iii / Paper 339 / 1

Each map $x\mapsto ⟨a_i,x⟩+b_i$ is an affine function. For $0\le t\le 1$, the pointwise maximum of convex functions satisfiesso $f$ is convex.

A vector $g$ is a subgradient of a convex function $f$ at $x$ whenfor every $y$. Choose any active index $j\in I(x):=\{i:f(x)=⟨a_i,x⟩+b_i\}$. Thenand hence $\boxed{a_j\in \partial f(x)}$. More generally, every convex combination of the active vectors is a subgradient, and in fact

Put $G={\max }_i\Vert a_i{\Vert }_2$. By the Cauchy-Schwarz inequality,Interchanging $x$ and $y$ proves the Lipschitz bound

The subgradient method chooses $g_k\in \partial f(x_k)$ and a step size $t_k>0$, then setsAssume, as the question's use of $\min f$ requires, that a minimizer $x_{\ast }$ exists, and write $R=\Vert x_0-x_{\ast }{\Vert }_2$. Since every subgradient here has Euclidean norm at most $G$, the standard best-iterate estimate isTaking a suitable constant step when the target accuracy is known, or a standard diminishing sequence, gives error at most $ϵ$ initerations, so the requested exponent is $p=2$.

Write $z_i(x)=⟨a_i,x⟩+b_i$. The log-sum-exp function is convex and composition with the affine functions $z_i$ preserves convexity, so $f_{\beta }$ is convex. Directly, its Hessian matrix will also be shown positive semidefinite in part d.

Let $M=f(x)={\max }_i{z}_i(x)$. Factoring $e^{\beta M}$ out of the sum givesAt least one term in the sum is $1$, while every term is at most $1$. Thereforeand henceThus $f_{\beta }$ is a uniform smooth maximum of the affine pieces of $f$.

Define the softmax weightsThey obey $p_i\ge 0$ and ${\sum }_i{p}_i=1$. The gradient is their weighted mean,Differentiating once more gives the covariance-form Hessian matrixFor every unit vector $u$,where $G={\max }_i\Vert a_i{\Vert }_2$. The Hessian is a covariance matrix, so it is positive semidefinite; the displayed upper bound also gives ${\nabla }^2{f}_{\beta }⪯\beta G^{2}I$ in the Loewner order. Consequently $f_{\beta }$ has a Lipschitz gradient with

Chooseso the smooth maximum error is at most $ϵ/2$ and the Lipschitz gradient constant isSuppose a minimizer of $f_{\beta }$ lies within distance $R$ of the starting point. The Nesterov accelerated gradient method can find $x$ such thatiniterations. If $x_{\ast }$ minimizes $f$, then the smoothing inequalities implyThis improves the nonsmooth subgradient method dependence from $O({ϵ}^{-2})$ to $O({ϵ}^{-1})$.