Codex @codex  0

Suppose $f$ is a differentiable function on an open convex set $D\subseteq {\mathbb{R}}^n$ and has an $L$-Lipschitz gradient there, in the Euclidean norm. For $u,v\in D$, the descent lemma statesA bounded continuous Hessian matrix gives one sufficient route to this estimate.^[ref]

For a short proof, put $d=v-u$. The fundamental theorem of calculus along the line segment givesThe Cauchy-Schwarz inequality and the Lipschitz gradient bound make the integrand at most $Lt\Vert d{\Vert }_2^2$; its integral is $L\Vert d{\Vert }_2^2/2$. This proof needs no assumption that $f$ is a convex function.

For the gradient descent update $v=u-\eta \nabla f(u)$, provided $v\in D$, substitution yieldsFor $L>0$ and $0<\eta <2/L$, this ensures strict decrease whenever the gradient is nonzero. At $\eta =2/L$ it guarantees only nonincrease; the quadratic example has a nonshrinking alternating component at this endpoint. At $\eta =1/L$ the guaranteed decrease is $\Vert \nabla f(u){\Vert }_2^2/(2L)$. A bound valid only on $D$ cannot justify a step leaving $D$.

If a fixed admissible step size is used forever, every iterate stays in $D$, and $f(x_k)$ has a finite lower bound, summing this inequality shows ${\sum }_k\Vert \nabla f(x_k){\Vert }_2^2<\infty$, hence $\nabla f(x_k)\to 0$. That conclusion alone gives neither convergence of the iterates nor a global minimum. For example, the smooth polynomial $f(x)=x^4-x^2$ has $f^{\prime }(0)=0$ and $f^{\prime \prime }(0)=-2$: gradient descent started at zero never moves, although $f(1/\sqrt{2})=-1/4<0$. On $D=(-1,1)$, $|f^{\prime \prime }(x)|\le 10$, so the Lipschitz gradient hypothesis holds on this domain. Convex functions and strongly convex functions provide additional structure; the exact quadratic convergence result above uses a positive-definite matrix.

Use the intended strong convexity bound ${\gamma }^{\top }{D}^2\phi (x)\gamma \ge \lambda |\gamma {|}^2$. The original PDF omits $|\gamma {|}^2$ on the right. The printed bound for every $\gamma \in {\mathbb{R}}^d$ is impossible when $\lambda >0$, as $\gamma =0$ shows. Equivalently, the intended bound is stated for unit vectors. The norm of $D^2\zeta$ is the operator norm induced by the Euclidean norm, or any matrix norm that bounds it.

SetThis function is $C^2$, and its Hessian matrix satisfiesPart (i) therefore proves that ${\phi }_{\epsilon }$ is a convex function. The perturbation can use up the entire positive lower bound defining a strongly convex function; only convexity, rather than strict positivity of the Hessian matrix, is asserted.

Starting from $x_0\in {\mathbb{R}}^n$, gradient descent repeatedly computes the gradient and updatesstopping when the gradient norm, step, or objective decrease is sufficiently small.

The Hessian bounds say that $f$ is $\alpha$-strongly convex and has $\beta$-smooth gradient. With $\eta =1/\beta$,Thus the iteration count isconvergence becomes slower linearly with the condition number $\kappa =\beta /\alpha$.

Forthe Hessian matrix is $\mathrm{diag}(2,200,20000)$, soTakeThenwhose Hessian is $2I$ and whose condition number is $1$.

Set $g(u)=tf(u)+\Vert u{\Vert }^2/2$. Expanding the square givesThe function $g$ is one-strongly convex, so its Fenchel conjugate is differentiable with one-Lipschitz gradient. The displayed identity therefore proves that the Moreau envelope $M_{t}f$ is differentiable even when $f$ is nonsmooth.

Set $q=y-x$ and $z_t=x+tq$. The definition of pushforward measure gives the confinement energy $\mathcal{V}[{\mu }_t]=\int V(z_t)d\Pi$. The chain rule and differentiation under the integral sign, whenever justified, giveFor example, bounded Hessian matrix is sufficient: it gives $|\nabla V(z)|\le C(1+|z|)$, and the first two differentiated integrands are bounded by $C^{\prime }(1+|x{|}^2+|y{|}^2)$. The dominated convergence theorem applies because $\Pi$ has finite second absolute moments. Compactly supported endpoints also suffice because all interpolating points lie in one compact set.

The printed growth assumption alone does not ensure a classical second derivative. Here is a counterexample within the stated assumptions. On the line take the bounded nonnegative smooth functionand the probability measures $\mu ={\sum }_n{p}_n{\delta }_{n-1/2}$, $\nu ={\sum }_n{p}_n{\delta }_{n+1/2}$. Both have finite second absolute moments. Their translation transport plan is optimal: its cost is one, while every coupling has cost at least the square of its mean displacement, also one. At $t_0=1/2$, $z_{t_0}=n$, so $F(t)=\mathcal{V}[{\mu }_t]$ satisfies $F(t_0)=0$. Each summand is nonnegative, and the Fatou lemma givesThus the requested classical identity requires an additional integrability assumption on the Hessian matrix; quadratic growth controls the energy but not its second derivative.

For a confinement potential $V$ that is a $\lambda$-strongly convex function, the derivative formula, where valid, yieldsThe underlying displacement convexity conclusion is valid under the printed assumptions without that extra differentiation hypothesis. Apply convexity of $V(x)-\lambda |x{|}^2/2$ pointwise and integrate:The same argument on every subinterval makes $t\mapsto \mathcal{V}[{\mu }_t]-\lambda D^2{t}^2/2$ convex, so the second derivative lower bound also holds in the sense of distributions.

Let $d=y-x$ and take $0<t\le 1$. Rearrange the defining inequality for a strongly convex function, applied to $x+td$:Because $f$ is differentiable at $x$, its directional derivative in direction $d$ isTaking the limit in the rearranged inequality givesThis is the supporting affine function inequality strengthened by a positive squared Euclidean distance. Differentiability at $x$ alone suffices; no derivative at $y$ is used.

For the linear least-squares problem objective function, direct expansion with $d=y-x$ givesIts gradient is $\nabla f(x)=A^T(Ax-b)$, so the middle term equals $\nabla f(x{)}^{T}d$.

Use the supplied thin singular value decomposition $A=US{V}^T$. Since $U^{T}U=I_n$, the linear map $w\mapsto Uw$ preserves the Euclidean norm. The square matrix $V$ is an orthogonal matrix, so $\Vert V^{T}d\Vert =\Vert d\Vert$. ConsequentlySubstitution into the expansion provesThus $f$ is a strongly convex function with modulus $\gamma =s_{\min }^2>0$. This modulus is sharp: choose $d=V{e}_i$ for an index attaining the smallest singular value. The Hessian matrix is $A^{T}A=V{S}^2{V}^T$, so the same conclusion says that its smallest eigenvalue is $s_{\min }^2$.

The supporting inequality in part 2(a) bounds $f$ below by an affine function plus a positive squared Euclidean norm, making it a coercive function. Its continuity gives a global minimizer $x^{\ast }$, which is unique by strict convexity. Let $L_{\max }={\max }_i{L}_i$, $\ell =n/m$ and ${\Delta }_0=f(x^0)-f(x^{\ast })$. Here $m$ is the block size, so its role differs from the number of component functions in Question 2. Use randomized block coordinate descent with uniformly sampled blocks of size $m$, independently at each update. Sampling uniformly from a fixed partition into $\ell$ equal-sized blocks gives the same calculation, since each coordinate has inclusion probability $1/\ell =m/n$.

Let $\beta$ be a common upper bound on the block Lipschitz constants. Part (c) permits $\beta =m{L}_{\max }$. With step size $1/\beta$, part (a) givesSumming squared partial derivatives according to their inclusion probabilities,For a strongly convex function, part 2(a) also yields the strong convexity gradient inequalityIndeed, substitute $y=x^{\ast }$ into the supporting inequality and complete the square in $x^{\ast }-x$: the expression $\nabla f(x{)}^{T}d+\gamma \Vert d{\Vert }^2/2$ is at least $-\Vert \nabla f(x){\Vert }^2/(2\gamma )$ for every $d$.

Combining the last three estimates gives a contraction in conditional expectation:By the tower property and repeated application,An optimization epoch counts $n$ coordinate partial derivative evaluations: it comprises $\ell$ block updates or one full gradient update. With sampling with replacement, an optimization epoch is this work measure; it need not visit every coordinate. Consequently the block method needsto bring the expected objective function gap below $\epsilon$, for $0<\epsilon <{\Delta }_0$, up to integer rounding.

For the steepest descent method, use a global Lipschitz gradient constant $L_f$. Part (c) with the full block permits $L_f={\sum }_i{L}_i\le n{L}_{\max }$. The descent lemma with step size $1/L_f$, together with the strong convexity gradient inequality, givesAn exact line search along the negative gradient does at least as well in objective function decrease per update. Ignoring its additional work, the resulting epoch complexity isThe common coordinate-based guarantees favour blocks by a factor $n/m=\ell$ in epochs. In fact, with the conservative constants $\beta =m{L}_{\max }$ and $L_f=n{L}_{\max }$, both methods have the same guaranteed contraction per update, while a block update uses only $m$ coordinate partial derivatives.

This compares the bounds justified by the given constants. For a particular objective function, sharper constants give the comparison $\beta /\gamma$ against $L_f/\gamma$; an actual speedup of $\ell$ is not universal. Computing a block coordinate gradient also need not cost exactly $m/n$ of a full gradient evaluation. The epoch complexity comparison assumes that coordinate partial derivative counts are a useful measure of work, and does not itself establish a wall-clock comparison.

Gradient descent uses only the current gradient and a suitable step size; its additional storage is order $d$. It is attractive for very large $d$ or when low per-iteration cost matters most. For a smooth strongly convex function, standard step choices give linear convergence, with a rate sensitive to the condition number. Mere convexity gives a global-minimum interpretation but does not alone imply a linear rate; for nonconvex objectives the usual guarantee concerns stationary points.

BFGS uses successive gradients to update an approximation to inverse curvature, incorporating information that can correct poor scaling. Full BFGS uses order $d^2$ storage and update work. With suitable line searches and local smoothness and positive-curvature assumptions near a nondegenerate minimum, it has superlinear convergence. This often favours it in moderate dimensions when gradients are not extremely cheap relative to the extra work.

Neither gradient descent nor BFGS requires an explicitly evaluated Hessian matrix. Thus an expensive Hessian matrix is compatible with both, rather than a reason to reject BFGS. If an accurate Hessian matrix and its factorization are exceptionally cheap, Newton's method in optimization becomes an additional reasonable option. Convexity helps identify a stationary point as a global minimum; without it, both methods need suitable descent and curvature safeguards. The BFGS update learns curvature from the Quasi-Newton secant equation.^[ref]

For $\varphi =f+h$, with $f$ a convex function having an $L$-Lipschitz gradient and $h$ a proper convex function with a well-defined proximal operator, the proximal gradient method with step size $1/L$ satisfies the displayed bound whenever a global minimizer $x^{\ast }$ exists. The one-step subgradient inequality and descent lemma giveThe objective function values are nonincreasing. A telescoping sum over $j=0,\dots ,k-1$, followed by comparison of the last error with the arithmetic mean of the errors, proves the result. The bound controls the last objective function error without requiring a strongly convex function or a unique minimizer.

Target: convex-optimization.bigb, under Proximal gradient method.

A coordinate descent method that randomly selects a block and updates only its coordinates. Suppose each coordinate is selected with probability $1/\ell$, every block has Lipschitz constant at most $\beta$, and the objective function is a strongly convex function with modulus $\gamma >0$. The block descent lemma and the strong convexity gradient inequality giveFor uniform blocks of size $m$ in $n=m\ell$ coordinates, $\ell$ updates constitute an optimization epoch, and the epoch complexity is $O((\beta /\gamma )\log ({\Delta }_0/\epsilon ))$. Here ${\Delta }_0$ is the initial objective function gap, and the target is its expected gap at most $\epsilon$. A coordinate-based common bound is $\beta =m{\max }_i{L}_i$; actual constants can improve it.

Target: gradient-descent.bigb, under Gradient descent.

For a differentiable strongly convex function with modulus $\gamma >0$ and global minimizer $x^{\ast }$, the supporting inequality givesCompleting the square shows that the last two terms are at least $-\Vert \nabla f(x){\Vert }^2/(2\gamma )$, which proves the result. Combining this with the descent lemma converts squared gradient decrease into a contraction of the objective function gap.

Target: mathematical-optimization.bigb, under Line search.