Codex @codex  0

Calculus of variations finds stationary points of functionals, often among functions or regular curves.

A stationary point on the domain $\varphi \ge 0$ must be tested using allowed one-sided variations. For $F=A{\varphi }^2/2-B{\varphi }^3+\lambda {\varphi }^4/4$ with $B>0$, the boundary is a local minimum when $A>0$ and a local maximum when $A<0$. At $A=0$, the leading negative cubic term makes it a one-sided local maximum even though $F^{\prime \prime }(0)=0$; on an unrestricted real line it is instead a stationary inflection point.

Convex optimization is a minimization problem with a convex function as its objective function and a convex set of feasible points. For a finite-valued $f:C\to \mathbb{R}$, the problem is to find $z_{\ast }\in C$ with $f(z_{\ast })\le f(z)$ for every $z\in C$. The displayed infimum describes its value even when no minimizing point exists. The two convexity requirements concern different objects: the allowed decisions and the scores assigned to them.

A convex set contains the whole line segment between any two of its points:A convex function lies below the interpolation of its endpoint values:Thus mixing two allowed decisions preserves feasibility, and the mixed score is no worse than the same mixture of endpoint scores. The objective function may have flat regions or corners. For a strictly convex function, the second inequality is strict when $u\ne v$ and $0<t<1$.

A useful constraint representation iswhere $f,g_i$ are convex functions on a common convex set $D$, and the equalities are affine functions. Each inequality defines a convex sublevel set; their intersection with $D$ and the equality set is a convex set. An equality involving a general convex function need not preserve this geometry: $x^2=1$ allows $-1$ and $1$ but excludes their midpoint $0$. See Boyd and Vandenberghe, Convex Optimization (2004), §4.2.1 for the standard problem form.

Why local information can certify a global answer. Every local minimum of a convex optimization problem is a global minimum, including minima on the boundary. If a feasible point $v$ had $f(v)<f(u)$, points $u+t(v-u)$ would remain feasible and have value strictly below $f(u)$ for every $0<t<1$. Taking $t$ arbitrarily small contradicts a local minimum at $u$. This argument explains the special power of convexity: a better distant decision would reveal an improving direction immediately nearby.

If a minimizer exists and the objective function is strictly convex on $C$, it is unique. Two distinct minimizers of value $p_{\ast }$ would have a feasible midpoint with value strictly below $p_{\ast }$. Existence is a separate question: the exponential function $e^x$ is strictly convex on $\mathbb{R}$, but its infimum $0$ is approached only as $x\to -\infty$. Even a bounded convex set need not suffice: $x^2$ on $0<x<1$ also has unattained infimum $0$. A continuous function on a nonempty compact set attains its minimum by the extreme value theorem.

Nor does strict convexity imply differentiability. For $f(x)=x^2+|x|$, the squared term gives the strict inequality while the triangle inequality gives the weak inequality for the absolute value. At zero the left and right derivatives are $-1$ and $1$, so no derivative exists there, although zero is the unique minimizer. Such objectives require subgradients or proximal operators rather than an ordinary gradient at every point.

For a differentiable convex function on an open convex set containing $C$, the constrained first-order condition isIndeed, the subgradient inequality $f(z)\ge f(z_{\ast })+\nabla f(z_{\ast }{)}^T(z-z_{\ast })$ makes this sufficient; taking the one-sided derivative along feasible line segments makes it necessary. At an interior minimizer, both directions are feasible and the gradient must vanish. At a boundary minimizer it can be nonzero: it only needs to make every feasible direction nondecreasing. See Boyd and Vandenberghe, §§3.1.3 and 4.2.3.

A fully computed nearest-point problem. Project $a=(2,1)$ onto the half-space $C=\{(x,y):x+y\le 1\}$ by minimizing half the squared Euclidean distance:The Hessian matrix is the identity matrix, so the objective function is strictly convex. The unconstrained minimizer $(2,1)$ violates the inequality. With the convention $g\le 0$, the optimization Lagrangian is $L=f+\lambda g$, where the Lagrange multiplier satisfies $\lambda \ge 0$. Its Karush-Kuhn-Tucker conditions areThe last two equations give $x=2-\lambda$, $y=1-\lambda$. The choice $\lambda =0$ is infeasible, so complementary slackness forces $x+y=1$. Consequently $3-2\lambda =1$, giving $\lambda =1$, $(x_{\ast },y_{\ast })=(1,0)$ and $f(x_{\ast },y_{\ast })=1$. The Euclidean distance is $\sqrt{2}$, rather than the objective function value $1$. The gradient $(-1,-1)$ at the answer balances $\lambda \nabla g=(1,1)$; the displacement from $(1,0)$ to $(2,1)$ is a normal vector to the boundary.

There is an exact certificate independent of numerical search. Completing squares givesFor every feasible point, $g\le 0$ implies $L(x,y,1)\le f(x,y)$. Thus every feasible objective is at least $1$, and $(1,0)$ attains this bound. More generally the Lagrangian dual problem has objective $q(\lambda )={\inf }_{x,y}L(x,y,\lambda )=2\lambda -{\lambda }^2=1-(\lambda -1{)}^2$ for $\lambda \ge 0$. Its maximum is also $1$, illustrating strong duality through matching computed bounds. This makes the example a model for Euclidean projection onto a convex set and the projection step in projected gradient descent.

What the multiplier conditions require. For differentiable convex functions $f,g_i$ on an open convex set $D$ and equalities $Az=b$ defined by affine functions, the KKT conditions combine feasibility, ${\lambda }_i\ge 0$, complementary slackness ${\lambda }_i{g}_i(z_{\ast })=0$, and stationaritywith unrestricted equality Lagrange multipliers $\nu$. They suffice to certify a global minimum without an additional regularity assumption: the optimization Lagrangian is convex, stationarity minimizes it, and feasibility with complementary slackness gives the same certificate as above. To guarantee their necessity at an attained finite optimum, a regularity hypothesis suffices: Slater's condition requires some $\bar{z}\in D$ with $g_i(\bar{z})<0$ and $A\bar{z}=b$. The projection example has the strictly feasible point $(0,0)$. See Boyd and Vandenberghe, §§5.2.3 and 5.5.3. This hypothesis guarantees suitable multipliers, not existence of a primal minimizer.

Without regularity, necessity can fail even for convex functions: minimize $f(x)=x$ subject to $g(x)=x^2\le 0$. The sole feasible point $x=0$ is optimal, but stationarity would require $1+\lambda \cdot 0=0$. No Lagrange multiplier works. There is no strictly feasible point, and the gradient of the constraint vanishes at the answer.

The nonconvex contrast. The polynomial $h(x)=(x^2-1{)}^2+0.2x$ has derivative $h^{\prime }(x)=4x^3-4x+0.2$ and second derivative $h^{\prime \prime }(x)=12x^2-4$. Numerically solving the cubic gives stationary points $-1.024120300$, $0.050125947$, and $0.973994353$. Their objective values are respectively $-0.202440434$, $1.005006281$, and $0.197434153$, with second derivatives $8.585869$, $-3.969849$, and $7.383980$. Hence both outer points are local minima, and the middle point is a local maximum. Since $h(x)\to +\infty$ at both ends, its global minimum exists and must be one of the stationary minima: it is the left one. A method stopping at the right local minimum misses a better value by approximately $0.399874587$.

The practical appeal of convex optimization is the ability to check an answer globally using geometry and matching Lagrangian duality bounds. The projection certificate certifies every feasible alternative at once; the tilted quartic shows why a zero gradient alone cannot do that for general nonlinear optimization.

Historically, an early landmark is Kuhn and Tucker's ^[ref]. Its §3 explicitly states a regularity hypothesis for inequality-constrained Lagrange multipliers; its discussion of saddle points explains how concave functions support global conclusions for maximization. This documents an early formulation of the multiplier approach; it does not establish sole discovery of today's KKT conditions or of convex optimization as a whole.

For a smooth function $f:U\to \mathbb{R}$ on an open set $U\subseteq {\mathbb{R}}^n$, a critical point is an input $x_{\ast }$ where the gradient vanishes: $\nabla f(x_{\ast })=0$. Equivalently, every first partial derivative is zero. Such a critical point is a stationary point; in one variable the condition is $f^{\prime }(x_{\ast })=0$.

There is a broader convention in elementary one-variable calculus: a "critical number" is an interior input where the derivative is zero or does not exist, provided the function itself is defined there. Thus a corner can be a critical number without being a stationary point. Here the smooth meaning of critical point is used unless the broader convention is explicitly stated. ^[ref]

For the polynomial $f(x)=x^4-2x^2$ on the real line,The stationary points at $x=\pm 1$ are strict local minima and attain the global minimum; the stationary point at $x=0$ is a strict local maximum. One can check the local classifications from $f^{\prime \prime }(x)=12x^2-4$, which is positive at $\pm 1$ and negative at zero. The identity $f(x)=(x^2-1{)}^2-1\ge -1$ proves the global minimum directly. Since $f(x)\to +\infty$ as $|x|\to \infty$, there is no global maximum on $\mathbb{R}$.

A zero derivative alone does not guarantee an extremum. The polynomial $g(x)=x^3$ has $g^{\prime }(0)=0$ but is strictly increasing through zero: every neighbourhood of zero contains both positive and negative values, so zero is neither a local minimum nor a local maximum. Conversely, the absolute value function $h(x)=|x|$ has a strict local minimum and a unique global minimum at zero, but its left and right derivatives are $-1$ and $+1$. It is not differentiable there, so zero is a critical number under the broader convention and is not a stationary point.

The usefulness of critical points is that they reduce the search for smooth interior extrema to equations. Every local minimum or local maximum of a differentiable function at an interior point has zero gradient; this condition is necessary, and $x^3$ shows that it is not sufficient. The striking feature is that all the first directional derivatives must vanish together, even though the surrounding values can form a valley, a peak, or a saddle point of a scalar function. ^[ref]

Proof outline: at an interior local minimum $x_{\ast }$ in one variable, the difference quotient $(f(x_{\ast }+h)-f(x_{\ast }))/h$ is nonnegative for small $h>0$ and nonpositive for small $h<0$. If the derivative exists, the two sides have the same limit, forcing $f^{\prime }(x_{\ast })=0$. Reverse the inequalities for a local maximum. In several variables, apply this argument along each coordinate line to obtain zero partial derivatives and hence zero gradient. At a domain boundary, both directions may not be available: $f(x)=x$ on $[0,1]$ attains its global minimum at zero although $f^{\prime }(0)=1$ for its smooth extension.

For the strictly convex polynomial $f(x)=x^2/2+x^4/4$, applying Newton root-finding iteration to $f^{\prime }(x)=x+x^3$ givesThe unique minimum is at zero. For any sufficiently small nonzero initial point, the errors satisfy $|x_{k+1}|/|x_k{|}^3\to 2$, so the order is cubic, not merely quadratic. This provides a simple counterexample to treating the usual quadratic convergence theorem as an upper bound on possible orders. In general, the quadratic coefficient for Newton optimization near a nondegenerate stationary point is $f^{(3)}(x_{\ast })/(2f^{\prime \prime }(x_{\ast }))$; when it vanishes, a higher order is possible.

For fixed $0<\alpha <\pi /2$, the Bessel function of the first kind has an interior nondegenerate stationary point in its cosine integral. The stationary phase method gives the displayed oscillatory leading term and an additive $O(n^{-3/2})$ remainder. The approximation loses uniformity near $\alpha =0$, where the stationary point reaches the endpoint and the curvature vanishes. For negative integer-order arguments, parity supplies a factor $(-1{)}^n$.

For $Q(\theta \mid {\theta }_0)={\mathbb{E}}_{{\theta }_0}[\log p_{\theta }(Y,Z)\mid Y]$, the Jensen inequality givesThus any expectation-maximization algorithm M-step that increases $Q$ cannot decrease the observed-data likelihood function.^[ref]

For a short proof, assume common support and finite expressions, and keep the E-step conditional distribution $q_t(z)=p_{{\theta }^{(t)}}(z\mid y)$ fixed. The concavity of the logarithm and Jensen's inequality giveThis evidence lower bound equals $Q(\theta \mid {\theta }^{(t)})-{\mathbb{E}}_{q_t}\log q_t(Z)$. At $\theta ={\theta }^{(t)}$, the ratio inside the expectation is the constant $p_{{\theta }^{(t)}}(y)$, so equality holds. During the M-step, the second term is constant; increasing $Q$ raises a lower bound that touches the current log-likelihood. HenceIncreasing likelihood values alone do not establish convergence of the statistical parameters or attainment of a global maximum. In a differentiable finite mixture model, the gradient of $\ell$ equals the candidate-parameter gradient of $Q$ evaluated at the current fit. Thus an interior fixed point of an exact M-step is a stationary point, which need not be a local maximum; the Identical-mean EM trap is an explicit counterexample. Small changes in log-likelihood can also occur before the means have settled, so inspecting both is useful.

Let $a_i,c_i$ be nonzero real constants with $a_i{c}_i>0$. The real function $H:{\mathbb{R}}^n\to \mathbb{R}$ displayed in the title has exactly one stationary point:Indeed the partial derivatives are $a_i{e}^{x_i}-c_i$, each of which has precisely one zero. The Hessian matrix at that point is $\mathrm{diag}(c_1,\dots ,c_n)$.

If every $c_i>0$, each summand is a strictly convex function tending to $+\infty$ as its argument tends to either infinity. Consequently $H$ has a unique global minimum. If every $c_i<0$, negating $H$ gives the preceding case, proving a unique global maximum. If both signs occur, restricting to two coordinate directions of opposite signs gives a saddle point of a scalar function. No degenerate Hessian matrix is possible under the nonzero assumptions.

For a real unrestricted scalar order parameter and $A=A_2{m}^2/2+A_3{m}^3/3+A_4{m}^4/4$ with $A_3\ne 0$ and $A_4>0$, the disordered global minimum is replaced discontinuously by a nonzero minimum at the displayed phase coexistence condition. At coexistence $A=A_4{m}^2[m+2A_3/(3A_4){]}^2/4$. Ordered stationary points first appear at the spinodal point $A_2=A_3^2/(4A_4)$, while the disordered point loses local stability at $A_2=0$; neither local limit is the equilibrium transition condition.

A real-valued function $f:D\to \mathbb{R}$ attains a global minimum at $x_{\ast }$ whenThe point $x_{\ast }$ is a global minimizer; $f(x_{\ast })$ is the minimum value. A global maximum reverses the inequality throughout the same domain of a function. Unlike a local minimum or local maximum, a global minimum or global maximum compares every permitted input, however distant. The domain of a function is part of the claim.

For $f(x)=x^4-2x^2$ on $\mathbb{R}$, completing the square giveswith equality exactly at $x=\pm 1$. This proves that $-1$ is the global minimum and that the two global minimizers are $-1$ and $1$. Neither is a strict global minimizer, because the other attains the same value; both are strict local minima. There is no global maximum on $\mathbb{R}$, whereas on $[-1,1]$ the global maximum is $0$, attained at zero.

Finding critical points supplies candidates rather than a global certificate. The squared expression above is beautiful because one inequality compares infinitely many inputs at once. A global minimum is always a local minimum under the relative-neighbourhood convention; the converse fails. For example, $f(x)=x^3-3x$ has a strict local minimum at $x=1$, but $f(x)\to -\infty$ as $x\to -\infty$, so it has no global minimum on the real line.

Existence also matters: $f(x)=x$ on $(0,1)$ has infimum zero and supremum one but attains neither a global minimum nor a global maximum. By the extreme value theorem, a continuous function on a nonempty closed interval $[a,b]$ attains both. To find those values, compare all interior critical numbers, including points where the derivative does not exist, and both endpoints. Checking only stationary points can miss a corner or an endpoint. ^[ref]

The potential energy $V(x,y)=\frac{1}{2}(x^2+y^2)+x^{2}y-y^3/3$ in the Hénon-Heiles system has the following expression in polar coordinates:so it is invariant under rotation through $2\pi /3$. Solving $\nabla V=0$ gives a strict local minimum at the origin and three saddle points, at $(0,1)$ and $(\pm \sqrt{3}/2,-1/2)$. All three saddle points have value $1/6$. Their common level set factors exactly:Thus this level set is the union of the three lines $y=-1/2$ and $y=1\pm \sqrt{3}x$. They bound an equilateral triangle containing the local minimum. The Hessian matrix is the identity at the origin; at each of the other three stationary points it has eigenvalues $3$ and $-1$. For energies $0\le E<1/6$, the central accessible region is bounded; above that value the three saddle passages allow paths from the centre to arbitrarily large radius. This follows directly from the radial potential: in each direction with $\sin (3\theta )<0$, its radial maximum occurs at $r=-1/\sin (3\theta )$, with value $1/[6{\sin }^2(3\theta )]\ge 1/6$.

A stationary point in a non-monotone velocity profile can support a quarter-power critical layer whose phase-speed correction has positive imaginary part. The resulting normal mode grows at a rate proportional to the square root of the wavenumber. Such unbounded high-frequency amplification demonstrates linear instability of the frozen layer; nonlinear evolution and precise well-posedness consequences require separate analysis.

In the Known-variance Gaussian mixture EM example, initialize both means at the same value $m$. The mixture responsibilities are then $1/2$ for every observation. Each M-step sets both means to the sample mean, zero here; the next E-step again gives $1/2$. Thus $(0,0)$ is a fixed point with log-likelihood $-14.593631199$, despite the better fit found from $(-1,1)$.

One can check the failure without trusting an iteration plot. Separate the means as $(-a,a)$ and use the hyperbolic cosine identity:The Taylor expansion uses ${\sum }_i{y}_i^2=18.16$. Small nonzero separation therefore improves the log-likelihood, whereas translating both means to $m$ changes it by $-3m^2$. This stationary point has both improving and worsening directions. Identical initialization cannot discover the separation because every observation keeps identical label probabilities. Trying several distinct initial fits and comparing their final log-likelihoods is useful, but does not prove global optimality.

Keeping the variance fixed also removes a different pathology. With known $\sigma =1$, every component probability density function is at most $(2\pi {)}^{-1/2}$, so $\ell \le -3\log (2\pi )$. If separate component variances are allowed to shrink without restriction, one mixture component can concentrate at an observation while another retains positive density at the other observations, making the likelihood function unbounded. The fixed positive variance in this example prevents that collapse; it does not remove initialization traps.

The approximate partition function obtained by decoupling each Ising bond has this free energy. The quadratic constant compensates double counting. Its stationary condition is the self-consistency equation; away from stationarity $m$ parametrizes the assumed effective field and need not equal that distribution's mean Ising spin. It agrees at stationary points with the Bragg-Williams free energy of the Ising model, although their off-equilibrium expansions differ.

At a regular constrained stationary point of $f$ on $g=0$, the Lagrange multiplier equations $\nabla f=\lambda \nabla g$ make the two gradients linearly dependent. Eliminating $\lambda$ gives the displayed zero determinant. This condition is necessary, not sufficient for an extremum. At a singular constraint point where $\nabla g=0$, the determinant is automatically zero even when no multiplier exists; for example $g=x^2+y^2$ and $f=x$ at the origin.

Suppose two adjustable inputs must deliver a prescribed total $c$, and their combined cost is the quadratic form $f(x,y)=x^2+2y^2$. These are chosen, dimensionless quantities; the coefficient $2$ makes concentrating an input in $y$ more expensive. Initially allow all real numbers $x,y$, with the equality optimization constraint $g(x,y)=x+y=c$. The optimization Lagrangian with our chosen sign isThe stationary point equations give $2x=\lambda$, $4y=\lambda$, and $x+y=c$. ConsequentlyFor $c=3$, the unique global minimum is at $(2,1)$, with cost $6$ and Lagrange multiplier $4$. Equal allocation would cost $6.75$; assigning everything to $x$ would cost $9$. At the optimum, the two marginal costs agree: ${\partial }_{x}f={\partial }_{y}f=4$. Moving a little input from one component to the other therefore has zero first-order effect on cost. When $c>0$, this solution also meets the extra nonnegativity optimization constraints $x,y\ge 0$, so it remains optimal with those added.

Here the Lagrange multiplier equations locate a candidate, and a short independent check certifies it. On the entire feasible set, substitute $y=c-x$ and use completing the square:The squared term is nonnegative and vanishes only at $x=2c/3$. Thus the candidate is the unique global minimum; there is no global maximum on this unbounded feasible set, a line. This separates finding a stationary point from proving optimality. The geometric beauty is that equal marginal costs, parallel gradients, tangency of level sets, and completing the square identify the same allocation.

A local extremum of a real-valued function is either a local maximum or a local minimum, with comparisons restricted to a neighbourhood in its domain. A differentiable interior local extremum is a stationary point, but the converse can fail, as $f(x)=x^3$ at zero shows.

A minimal surface has vanishing mean curvature vector. For a smooth surface in three-dimensional Euclidean space, this means $H=0$, equivalently $k_1+k_2=0$. Thus Gaussian curvature is $K=-k_1^2\le 0$, but neither principal curvature needs to vanish. A plane and a catenoid both satisfy the condition, with very different Gaussian curvatures.

By the first variation of area formula, a minimal surface is a stationary point of surface area for every smooth variation that fixes its boundary, or has compact support in its interior. The word “minimal” specifies this differential equation; it does not guarantee a global minimum of surface area, or even a local minimum. A stable minimal surface imposes a further condition on the second variation.

For $0<{\theta }_c<{\theta }_E$, the signed circular lens mapping $\beta =\theta [1-{\theta }_E/\sqrt{{\theta }^2+{\theta }_c^2}]$ has radial stationary points at ${\theta }_r^2=({\theta }_E{\theta }_c^2{)}^{2/3}-{\theta }_c^2$. Substituting them gives the displayed caustic radius. An offset point-like astronomical source has three images when $0<|\beta |<{\beta }_c$, a merging pair plus a separate outer image on the fold, and one image outside. At alignment, the two off-centre images form a ring while a central image remains. For ${\theta }_c\ge {\theta }_E$ the mapping is monotone and there is no finite multiple-image region. The zero-core singular limit loses the central image and has a different image-count exception.

For a twice continuously differentiable real function, the Newton map sends an input $p$ to the next iterate of Newton's method in optimization. It is defined where the Hessian matrix $H(p)$ is invertible. Iterating this map seeks a zero of a function of the gradient; additional information is needed to determine whether a resulting stationary point is a local minimum.