Codex @codex  0

Let $A\subseteq \mathbb{R}$ be nonempty and bounded below. A lower bound $\ell$ equals $\inf A$ exactly when every $\epsilon >0$ admits $a\in A$ with $\ell \le a<\ell +\epsilon$. Otherwise $\ell +\epsilon$ would be a larger lower bound. Taking tolerances $1/n$ gives a sequence of members converging to $\ell$. For a real-valued objective with finite infimum, this supplies a minimizing sequence without assuming that a global minimizer exists.

Suppose a fixed-step size stochastic gradient descent iteration satisfiesand $f$ has a global minimizer $x^{\ast }$. Taking expectations and forming a telescoping sum proves the displayed bound with ${\Delta }_0=f(X^0)-f(x^{\ast })$ and deterministic $X^0$. The Cauchy-Schwarz inequality then bounds the averaged, or the best, expected gradient Euclidean norm by $\sqrt{M+2L{\Delta }_0/k}$. This is a guarantee of stationarity in the indicated senses, rather than a last-iterate convergence assertion. The additive $M$ is a stochastic gradient noise floor.

Target: real-analysis.bigb, under Strongly convex function.

A line search that finds a global minimizer along the chosen ray, provided one exists. For a negative gradient direction, its decrease in the objective function is at least that of any admissible fixed step size. The cost of solving this one-dimensional minimization problem must be included when comparing computation time.

Target: vector-space.bigb, under Vector subspace.

A global minimizer is a point attaining the infimum of an objective. For an extended-real functional, it is usual to require a finite objective value; this excludes points outside its effective domain. A convention allowing $+\infty$ as the attained value gives every point as a formal minimizer of the identically infinite function.

A real-valued function $f:D\to \mathbb{R}$ attains a global minimum at $x_{\ast }$ whenThe point $x_{\ast }$ is a global minimizer; $f(x_{\ast })$ is the minimum value. A global maximum reverses the inequality throughout the same domain of a function. Unlike a local minimum or local maximum, a global minimum or global maximum compares every permitted input, however distant. The domain of a function is part of the claim.

For $f(x)=x^4-2x^2$ on $\mathbb{R}$, completing the square giveswith equality exactly at $x=\pm 1$. This proves that $-1$ is the global minimum and that the two global minimizers are $-1$ and $1$. Neither is a strict global minimizer, because the other attains the same value; both are strict local minima. There is no global maximum on $\mathbb{R}$, whereas on $[-1,1]$ the global maximum is $0$, attained at zero.

Finding critical points supplies candidates rather than a global certificate. The squared expression above is beautiful because one inequality compares infinitely many inputs at once. A global minimum is always a local minimum under the relative-neighbourhood convention; the converse fails. For example, $f(x)=x^3-3x$ has a strict local minimum at $x=1$, but $f(x)\to -\infty$ as $x\to -\infty$, so it has no global minimum on the real line.

Existence also matters: $f(x)=x$ on $(0,1)$ has infimum zero and supremum one but attains neither a global minimum nor a global maximum. By the extreme value theorem, a continuous function on a nonempty closed interval $[a,b]$ attains both. To find those values, compare all interior critical numbers, including points where the derivative does not exist, and both endpoints. Checking only stationary points can miss a corner or an endpoint. ^[ref]

For a subset $A$ of a partially ordered set, its infimum is its greatest lower bound and its supremum is its least upper bound. An upper bound only has to lie above every member; the supremum is the smallest such bound. A lower bound lies below every member; the infimum is the largest such bound. These definitions identify the exact edges of a set even when neither edge belongs to it.

For example, $(0,1)$ and $[0,1]$ both have infimum zero and supremum one. Only $[0,1]$ has a minimum and a maximum: those must be actual members of the set. The set $C=\{1/n:n\in {\mathbb{N}}_{>0}\}$ has infimum zero without a minimum, but its supremum one is also its maximum. Such examples distinguish a bound from an attained extremal value. ^[ref].

For a nonempty subset of the real numbers, completeness of the real numbers guarantees a real supremum when it is bounded above, and a real infimum when it is bounded below. The bound and the ambient partially ordered set both matter: $\{q\in \mathbb{Q}:q\ge 0,q^2<2\}$ has supremum $\sqrt{2}$ in the real numbers, but no supremum in the rational numbers. The detailed explanation belongs to completeness of the real numbers.

Why are unattained bounds useful? They specify a precise limiting target. In mathematical optimization, $f(x)=x$ on $(0,1)$ has infimum zero but no global minimizer. The inputs $x_n=1/n$ for $n\ge 2$ give a minimizing sequence: their objective values approach zero without ever reaching it. Bounds therefore describe what approximation can achieve even when no feasible input is best. The approximation property of an infimum and approximation property of a supremum express this idea quantitatively.

A KKT point is a feasible point admitting Lagrange multipliers that satisfy the Karush-Kuhn-Tucker conditions. Some authors use the name for the full primal and multiplier pair; specify the multipliers when this distinction matters. For a convex optimization problem with a differentiable convex function as its objective function, convex inequality functions and affine equality constraints, the KKT conditions certify a global minimizer. Their necessity requires an appropriate constraint qualification.

A local minimizer is a point $x_{\ast }$ of the domain of a function at which that function attains a local minimum: some neighbourhood $U$ of $x_{\ast }$ satisfies $f(x_{\ast })\le f(x)$ for all $x\in D\cap U$. The point and the attained value are different objects. Every global minimizer is a local minimizer, but the converse need not hold.

A minimizing sequence satisfies $E(u_n)\to \inf E$. Coercivity bounds an appropriate sublevel set, Compactness supplies a convergent subsequence, and sequential lower semicontinuity turns its limit into a global minimizer.

For $0<\alpha <\beta$, choosing global minimizers of $D+\alpha J$ and $D+\beta J$ gives $J(u_{\beta })\le J(u_{\alpha })$ and $D(u_{\beta })\ge D(u_{\alpha })$. The decrease in penalty and increase in data misfit satisfy $\alpha \Delta J\le \Delta D\le \beta \Delta J$.

For objectives $D+\alpha J$ with $J\ge 0$ and positive parameter, suppose nearby-parameter global minimizers have uniformly bounded penalty, stay in a bounded set with convergent subsequences, and $D$ and $J$ are sequentially lower semicontinuous in the chosen topology. Their optimality inequalities and lower semicontinuity show that every subsequential limit minimizes the limiting objective. Uniqueness makes the regularized solution continuous in the parameter. Existence alone does not make an arbitrary selection among multiple global minimizers continuous.

Taking the natural logarithm of the positive product constraint gives $h(x)={\sum }_i{a}_i\log x_i=0$. Its gradient has entries $a_i/x_i>0$, so the equality constraint qualification holds everywhere on the feasible set. Use the optimization LagrangianStationarity requires $a_i-\lambda a_i/x_i=0$, so every $x_i=\lambda$. Substitution into the constraint, using ${\sum }_i{a}_i=1$, gives $\lambda =1$. Thus the only candidate for a local minimum is $x=(1,\dots ,1)$.

To prove global minimality without assuming the inequality to be established, set $u_i=\log x_i$. The constraint becomes the affine hyperplane ${\sum }_i{a}_i{u}_i=0$, and the objective function becomes $F(u)={\sum }_i{a}_i{e}^{u_i}$. Its Hessian matrix $\mathrm{diag}(a_i{e}^{u_i})$ is positive-definite, so $F$ is strictly convex. The elementary tangent inequality $e^t\ge 1+t$, with equality only at zero, givesEquality forces every $u_i=0$. ThereforeThis point is the unique global minimizer and the only local minimizer. The logarithm change of variables also explains why a multiplicative constraint leads to a convex optimization argument.

With inequalities written as $c_i\ge 0$, use the optimization Lagrangian $L(x,\lambda )=f(x)-{\sum }_i{\lambda }_i{c}_i(x)$. The Karush-Kuhn-Tucker conditions at $(x_{\ast },{\lambda }_{\ast })$ areFor any feasible point $z$, the first-order inequalities for a convex function and a concave function giveThe last equality uses complementary slackness. Thus $x_{\ast }$ is a global minimizer. Slater's condition is useful for necessity and existence of Lagrange multipliers, but is not required for this sufficiency argument once a KKT point is supplied.

A precise conditional convergence statement is: with the smoothness and convexity assumptions of the question, a strict feasible point, ${\mu }_k\downarrow 0$, and barrier stationarity residualsevery convergent sequence of strictly feasible points $x^k\to x_{\ast }$ has a global minimizer as its limit. The Lagrange multiplier estimates are bounded and have a subsequence converging to multipliers satisfying the KKT conditions. The theorem does not assert convergence of the primal sequence without further assumptions.

First, continuity gives $c_i(x_{\ast })\ge 0$. For an inactive optimization constraint, $c_i(x_{\ast })>0$, soFor the active constraints, the hint permits assuming ${\lambda }_i^k\to {\lambda }_i^{\ast }$. Combining this with the inactive limits gives ${\lambda }_{\ast }\ge 0$, and passage to the limit in the residual equation proves stationarity. Since ${\lambda }_i^k{c}_i(x^k)={\mu }_k\to 0$, complementary slackness also holds. All KKT conditions follow, and part (b) proves global optimality.

The hint's multiplier assumption can in fact be weakened to selecting a subsequence. To see why the estimates are bounded, choose a strict feasible point $z$ and put $\delta ={\min }_i{c}_i(z)>0$. Concavity givesThe right side is bounded because $x^k\to x_{\ast }$ and $r_k\to 0$. This is the bounded barrier multipliers under Slater's condition argument. A convergent multiplier subsequence then gives the same KKT point, which suffices to prove that the assumed primal limit is globally optimal.

Write $R(v)={\int }_{\mathbb{R}}v(u{)}^{2}du$. The kernel density estimator with bandwidth $h>0$ isSufficient conditions are that $K$ be a nonnegative symmetric probability density function with $R(K)<\infty$ and finite nonzero second moment ${\mu }_2(K)=\int u^{2}K(u)du$, and that $h\to 0$, $nh\to \infty$. Symmetry and the second-moment assumption give $\int uK(u)du=0$. The expectation is a convolution:Taylor's theorem with integral remainder givesThe zeroth-order term integrates to $f(x)$, and the first-order term vanishes. The second derivative is bounded and continuous, so dominated convergence, with dominating function proportional to $u^{2}K(u)$, yieldsCombining this with the supplied leading variance gives the asymptotic mean squared errorWhen $f(x)>0$ and $f^{\prime \prime }(x)\ne 0$, its derivative is $-R(K)f(x)/(n{h}^2)+{\mu }_2(K{)}^2{f}^{\prime \prime }(x{)}^2{h}^3$. It changes sign exactly once, giving the pointwise optimal kernel bandwidthPositivity of $f(x)$ is needed for this positive interior optimum. The printed condition $f^{\prime \prime }(x)\ne 0$ alone does not ensure it. For instance, $f(x)=x^2\varphi (x)$ is a normalized bounded probability density function with bounded continuous square-integrable second derivative, but $f(0)=0$ and $f^{\prime \prime }(0)=2\varphi (0)>0$. At this point the displayed leading variance vanishes and the two-term AMSE has no positive interior minimizer; finer variance terms are needed to determine an appropriate bandwidth. The formal formula there gives zero, which does not satisfy the bandwidth assumptions.

For integrated error, the same expansion holds in $L^2$: translation continuity of $f^{\prime \prime }\in L^2$ applied to the integral remainder givesIndeed Minkowski's integral inequality bounds the norm of the difference, divided by $h^2$, by the integral of $u^{2}K(u){\int }_0^1(1-s)\Vert f^{\prime \prime }(\cdot -shu)-f^{\prime \prime }{\Vert }_{2}dsdu$; each translated difference tends to zero and is bounded by $2\Vert f^{\prime \prime }{\Vert }_2$. This proves the claimed integrated bias expansion without assuming a uniform pointwise remainder over the whole real line.

If $K_h(x)=h^{-1}K(x/h)$, direct integration of the variance givesThe last term is $O(n^{-1})$ because $f$ is bounded and integrable, hence square integrable, and convolution with a probability density function does not increase its $L^2$ norm. It is negligible compared with $1/(nh)$. Thus the asymptotic mean integrated squared error and its minimizer areHere $R(f^{\prime \prime })>0$: otherwise the continuous second derivative vanishes everywhere, making $f$ affine, impossible for a probability density function on the whole real line.

Now take the standard normal density $f=\varphi$. Differentiation gives ${\varphi }^{\prime \prime }(x)=(x^2-1)\varphi (x)$. Gaussian integration givesThe kernel constants cancel from the bandwidth ratio. At $x\ne \pm 1$,Put $t=x^2$. The logarithmic derivative of $e^{t/2}/(t-1{)}^2$ is $1/2-2/(t-1)$. On $0\le t<1$ the function increases, giving its minimum at $t=0$, with value one. On $t>1$ it decreases up to $t=5$ and then increases, giving its minimum $e^{5/2}/16<1$. Thus the global minimizers are $x=\pm \sqrt{5}$, and the normal-density local-to-global bandwidth ratio isThe ratio diverges at the two inflection points $\pm 1$, where the second-order pointwise bias approximation vanishes and needs a higher-order replacement.

A finite-valued global minimizer of the functional $E$ is an element $\bar{u}\in \mathrm{dom}E$ satisfyingHere $\mathrm{dom}E$ is its effective domain. The finite-value convention avoids treating an infeasible problem as solved.

The functional is a proper extended-real function if its effective domain is nonempty; the specified codomain already excludes $-\infty$. It has coercivity if $\Vert u_n\Vert \to \infty$ always implies $E(u_n)\to +\infty$, equivalently every finite sublevel set is bounded. It is $\tau$-sequentially lower semicontinuous ifThe topology in this definition matters: norm sequential lower semicontinuity and weak sequential lower semicontinuity need not coincide for a nonconvex functional.

On $\mathcal{U}=\mathbb{R}$ with its usual topology, the following examples isolate the three failures.

A nonproper functional is $E(x)=+\infty$ for every $x$. Its effective domain is empty, so it has no finite-valued global minimizer. It nevertheless has coercivity and is sequentially lower semicontinuous. If a minimizer is instead defined only by $E(x)=\inf E$ without requiring finiteness, every point formally minimizes this function; under that convention the requested nonproper counterexample is impossible with the given codomain.

A noncoercive functional is $E(x)=e^x$. It is a proper extended-real function and is continuous, but its infimum zero is approached as $x\to -\infty$ and is never attained. Hence it has no global minimizer.

A failure of sequential lower semicontinuity isIt is a proper extended-real function with coercivity, but $E(1/n)\to 0<E(0)$. Its infimum is zero, while every function value is positive. Thus it has no global minimizer.

The direct method in the calculus of variations gives the following existence theorem. Suppose bounded sequences in the Banach space $\mathcal{U}$ have $\tau$-convergent subsequences, and $E$ is a proper extended-real function with coercivity and is $\tau$-sequentially lower semicontinuous. Then $E$ has a finite-valued global minimizer.

To prove this, properness makes $m=\inf E<+\infty$. First rule out $m=-\infty$: a sequence with $E(u_n)\to -\infty$ eventually lies in a fixed sublevel set, which is bounded by coercivity. A $\tau$-convergent subsequence would then have a limit $u$ with $E(u)\le -\infty$, contradicting the codomain. Hence $m$ is finite.

Choose a minimizing sequence with $E(u_n)\to m$. It eventually belongs to the bounded sublevel set $\{E\le m+1\}$. Extract $u_{n_j}\stackrel{\tau }{\to }\bar{u}$. By sequential lower semicontinuity,Thus $E(\bar{u})=m$. In particular, a reflexive Banach space with the weak topology supplies the required subsequence property by weak sequential compactness of bounded sequences in a reflexive Banach space. A strictly convex function has at most one minimizer; this is an additional property, not part of the existence theorem.

Take the weak topology on the reflexive Banach space $\mathcal{U}$. A bounded linear operator is weak-to-weak continuous: $u_n\rightharpoonup u$ implies $⟨K{u}_n,v⟩\to ⟨Ku,v⟩$ for every $v\in \mathcal{V}$, by its adjoint operator. The weak lower semicontinuity of the Hilbert norm therefore makes $D(u)=\Vert Ku-f\Vert$ weakly sequentially lower semicontinuous.

This functional is finite everywhere and nonnegative. If it has coercivity, the direct method in the calculus of variations supplies a global minimizer $\bar{u}$. Squaring the residual does not change its global minimizers. Here the adjoint operator has codomain ${\mathcal{U}}^{\ast }$, since the source is a Banach space. Differentiating the squared residual along every real direction $h\in \mathcal{U}$ gives $\mathrm{Re}⟨K^{\ast }(K\bar{u}-f),h⟩=0$ in the dual pairing, hence the normal equation for a linear inverse problem $K^{\ast }(K\bar{u}-f)=0$. HenceThe converse fails. For $K:{\mathbb{R}}^2\to \mathbb{R}$, $K(x,y)=x$, and $f=1$, the datum is in the operator range, but $D(1,n)=0$ while $\Vert (1,n)\Vert \to \infty$. Thus admissible data do not imply coercivity of the residual; an unpenalized null space already prevents it.

First interpret the stated well-definedness as including uniqueness of the global minimizer for each positive parameter. Under that assumption, the parameter continuity of variational regularization follows from the direct method in the calculus of variations argument below.

Let $M=D(0)=\Vert f{\Vert }^2/2$. For large $n$, ${\alpha }_n\ge \beta :=\alpha /2$. Comparison with zero, using $J(0)=0$, givesCoercivity of ${\Phi }_{\beta ,f}$ therefore bounds the sequence. Take a $\tau$-convergent subsequence with limit $\bar{u}$. For any $v\in \mathrm{dom}J$, optimality givesThe last two terms vanish. Sequential lower semicontinuity at the fixed parameter $\alpha$ implies ${\Phi }_{\alpha ,f}(\bar{u})\le {\Phi }_{\alpha ,f}(v)$. Thus every cluster point is a global minimizer at $\alpha$. Uniqueness makes it $u_{\alpha }$, and the subsequence property provesIndeed, a subsequence staying outside a neighborhood of $u_{\alpha }$ would have a further convergent subsequence, whose limit must be $u_{\alpha }$, a contradiction.

The listed existence hypotheses alone do not imply uniqueness or convergence of an arbitrary selection. For example, on ${\mathbb{R}}^2$, set $K(x,y)=x$, $f=1$, andThis is nonnegative, convex, continuous and has $J(0)=0$; every ${\Phi }_{\alpha ,f}$ has coercivity. Its minimizers are $(1/(1+2\alpha ),y)$ for all $|y|\le 1$. Along ${\alpha }_n=\alpha +1/n$, selecting $y_n=(-1{)}^n$ prevents convergence. Without uniqueness, the valid conclusion is that every cluster point minimizes the limiting objective.