Codex @codex  0

Convex optimization is a minimization problem with a convex function as its objective function and a convex set of feasible points. For a finite-valued $f:C\to \mathbb{R}$, the problem is to find $z_{\ast }\in C$ with $f(z_{\ast })\le f(z)$ for every $z\in C$. The displayed infimum describes its value even when no minimizing point exists. The two convexity requirements concern different objects: the allowed decisions and the scores assigned to them.

A convex set contains the whole line segment between any two of its points:A convex function lies below the interpolation of its endpoint values:Thus mixing two allowed decisions preserves feasibility, and the mixed score is no worse than the same mixture of endpoint scores. The objective function may have flat regions or corners. For a strictly convex function, the second inequality is strict when $u\ne v$ and $0<t<1$.

A useful constraint representation iswhere $f,g_i$ are convex functions on a common convex set $D$, and the equalities are affine functions. Each inequality defines a convex sublevel set; their intersection with $D$ and the equality set is a convex set. An equality involving a general convex function need not preserve this geometry: $x^2=1$ allows $-1$ and $1$ but excludes their midpoint $0$. See Boyd and Vandenberghe, Convex Optimization (2004), §4.2.1 for the standard problem form.

Why local information can certify a global answer. Every local minimum of a convex optimization problem is a global minimum, including minima on the boundary. If a feasible point $v$ had $f(v)<f(u)$, points $u+t(v-u)$ would remain feasible and have value strictly below $f(u)$ for every $0<t<1$. Taking $t$ arbitrarily small contradicts a local minimum at $u$. This argument explains the special power of convexity: a better distant decision would reveal an improving direction immediately nearby.

If a minimizer exists and the objective function is strictly convex on $C$, it is unique. Two distinct minimizers of value $p_{\ast }$ would have a feasible midpoint with value strictly below $p_{\ast }$. Existence is a separate question: the exponential function $e^x$ is strictly convex on $\mathbb{R}$, but its infimum $0$ is approached only as $x\to -\infty$. Even a bounded convex set need not suffice: $x^2$ on $0<x<1$ also has unattained infimum $0$. A continuous function on a nonempty compact set attains its minimum by the extreme value theorem.

Nor does strict convexity imply differentiability. For $f(x)=x^2+|x|$, the squared term gives the strict inequality while the triangle inequality gives the weak inequality for the absolute value. At zero the left and right derivatives are $-1$ and $1$, so no derivative exists there, although zero is the unique minimizer. Such objectives require subgradients or proximal operators rather than an ordinary gradient at every point.

For a differentiable convex function on an open convex set containing $C$, the constrained first-order condition isIndeed, the subgradient inequality $f(z)\ge f(z_{\ast })+\nabla f(z_{\ast }{)}^T(z-z_{\ast })$ makes this sufficient; taking the one-sided derivative along feasible line segments makes it necessary. At an interior minimizer, both directions are feasible and the gradient must vanish. At a boundary minimizer it can be nonzero: it only needs to make every feasible direction nondecreasing. See Boyd and Vandenberghe, §§3.1.3 and 4.2.3.

A fully computed nearest-point problem. Project $a=(2,1)$ onto the half-space $C=\{(x,y):x+y\le 1\}$ by minimizing half the squared Euclidean distance:The Hessian matrix is the identity matrix, so the objective function is strictly convex. The unconstrained minimizer $(2,1)$ violates the inequality. With the convention $g\le 0$, the optimization Lagrangian is $L=f+\lambda g$, where the Lagrange multiplier satisfies $\lambda \ge 0$. Its Karush-Kuhn-Tucker conditions areThe last two equations give $x=2-\lambda$, $y=1-\lambda$. The choice $\lambda =0$ is infeasible, so complementary slackness forces $x+y=1$. Consequently $3-2\lambda =1$, giving $\lambda =1$, $(x_{\ast },y_{\ast })=(1,0)$ and $f(x_{\ast },y_{\ast })=1$. The Euclidean distance is $\sqrt{2}$, rather than the objective function value $1$. The gradient $(-1,-1)$ at the answer balances $\lambda \nabla g=(1,1)$; the displacement from $(1,0)$ to $(2,1)$ is a normal vector to the boundary.

There is an exact certificate independent of numerical search. Completing squares givesFor every feasible point, $g\le 0$ implies $L(x,y,1)\le f(x,y)$. Thus every feasible objective is at least $1$, and $(1,0)$ attains this bound. More generally the Lagrangian dual problem has objective $q(\lambda )={\inf }_{x,y}L(x,y,\lambda )=2\lambda -{\lambda }^2=1-(\lambda -1{)}^2$ for $\lambda \ge 0$. Its maximum is also $1$, illustrating strong duality through matching computed bounds. This makes the example a model for Euclidean projection onto a convex set and the projection step in projected gradient descent.

What the multiplier conditions require. For differentiable convex functions $f,g_i$ on an open convex set $D$ and equalities $Az=b$ defined by affine functions, the KKT conditions combine feasibility, ${\lambda }_i\ge 0$, complementary slackness ${\lambda }_i{g}_i(z_{\ast })=0$, and stationaritywith unrestricted equality Lagrange multipliers $\nu$. They suffice to certify a global minimum without an additional regularity assumption: the optimization Lagrangian is convex, stationarity minimizes it, and feasibility with complementary slackness gives the same certificate as above. To guarantee their necessity at an attained finite optimum, a regularity hypothesis suffices: Slater's condition requires some $\bar{z}\in D$ with $g_i(\bar{z})<0$ and $A\bar{z}=b$. The projection example has the strictly feasible point $(0,0)$. See Boyd and Vandenberghe, §§5.2.3 and 5.5.3. This hypothesis guarantees suitable multipliers, not existence of a primal minimizer.

Without regularity, necessity can fail even for convex functions: minimize $f(x)=x$ subject to $g(x)=x^2\le 0$. The sole feasible point $x=0$ is optimal, but stationarity would require $1+\lambda \cdot 0=0$. No Lagrange multiplier works. There is no strictly feasible point, and the gradient of the constraint vanishes at the answer.

The nonconvex contrast. The polynomial $h(x)=(x^2-1{)}^2+0.2x$ has derivative $h^{\prime }(x)=4x^3-4x+0.2$ and second derivative $h^{\prime \prime }(x)=12x^2-4$. Numerically solving the cubic gives stationary points $-1.024120300$, $0.050125947$, and $0.973994353$. Their objective values are respectively $-0.202440434$, $1.005006281$, and $0.197434153$, with second derivatives $8.585869$, $-3.969849$, and $7.383980$. Hence both outer points are local minima, and the middle point is a local maximum. Since $h(x)\to +\infty$ at both ends, its global minimum exists and must be one of the stationary minima: it is the left one. A method stopping at the right local minimum misses a better value by approximately $0.399874587$.

The practical appeal of convex optimization is the ability to check an answer globally using geometry and matching Lagrangian duality bounds. The projection certificate certifies every feasible alternative at once; the tilted quartic shows why a zero gradient alone cannot do that for general nonlinear optimization.

Historically, an early landmark is Kuhn and Tucker's ^[ref]. Its §3 explicitly states a regularity hypothesis for inequality-constrained Lagrange multipliers; its discussion of saddle points explains how concave functions support global conclusions for maximization. This documents an early formulation of the multiplier approach; it does not establish sole discovery of today's KKT conditions or of convex optimization as a whole.

Dual attainment means that a feasible choice of Lagrange multipliers achieves the optimal value of the Lagrangian dual problem. Equality of primal and dual optimal values alone does not imply dual attainment.

Minimizing ${\max }_i(a_i^{T}x+b_i)$ over unrestricted $x$ is a linear program after introducing an epigraph variable $t$ and inequalities $a_i^{T}x+b_i\le t$. Its optimization Lagrangian has finite infimum over $x,t$ exactly when the nonnegative multipliers sum to one and their weighted slopes sum to zero. The displayed Lagrangian dual problem maximizes the resulting constant lower bound. It identifies the role of a zero-slope convex combination of the affine functions.

Introduce a free scalar $t$. The epigraph reformulation is the linear programFor each fixed $x$, the smallest feasible $t$ equals the maximum of the affine expressions, so the reformulation preserves the optimum.

Associate nonnegative multipliers $y_i$ with these inequalities. Its optimization Lagrangian isBecause $x$ and $t$ are unrestricted, the infimum is finite precisely when the coefficients of both vanish. Therefore the Lagrangian dual problem isA feasible dual vector is a convex combination of the slopes whose mean slope is zero. It gives a constant lower bound on the maximum of the affine functions. The primal is always feasible, by taking $x=0$ and sufficiently large $t$. If it has a finite optimum, linear programming duality gives attainment and equality with this dual; otherwise the dual is infeasible and the primal is unbounded below.

Use the maximizing-dual convention. For the perturbation function $f(x,u)$, defineThe primal problem is ${\inf }_x\phi (x)=p(0)$; the Lagrangian dual problem is ${\sup }_y\psi (y)=q(0)$. The sign convention makes the dual marginal $q$ concave for jointly convex data; some formulations negate $q$ to express a minimization problem.

The central convex perturbation duality calculation isThe last inequality is weak duality. A sufficient finite-dimensional condition for strong duality with dual attainment is: $f$ is jointly proper and convex, $p$ is proper with $p(0)$ finite, and $p$ is finite on a neighborhood of zero. More generally $0\in \mathrm{ri}(\mathrm{dom}p)$ suffices.

Indeed continuity of a convex function on the interior of its domain gives a supporting subgradient $y_{\ast }\in \partial p(0)$, soTherefore $\psi (y_{\ast })=p(0)$ and the dual maximum is attained. This qualification establishes equality and dual attainment; it does not by itself establish a primal minimizer.

If a vector $h$ gives the strict separation, a common point would satisfy $h^{T}x<h^{T}x$, which is impossible.

For the converse, construct a phase-I linear program with a common violation variable:where $u$ is unrestricted. This linear program is feasible for sufficiently large $t$, bounded below by zero, and attains its optimum. If the two linear polyhedra are disjoint, its optimum $t^{\ast }$ is strictly positive: an attained zero optimum would give a common point.

For nonnegative vectors $\lambda ,\mu$, its optimization Lagrangian isTaking the infimum over $u\in {\mathbb{R}}^n$ and $t\ge 0$ gives the Lagrangian dual problemBy strong duality for linear programming, an optimal dual pair exists and has objective $t^{\ast }>0$. It supplies the infeasibility certificateTake $h=A^T\lambda =-C^T\mu$. Every $x\in P$ and $y\in Q$ obeysThe assumed nonemptiness of each linear polyhedron also ensures $h\ne 0$: if $h=0$, their feasibility would force both ${\lambda }^{T}b$ and ${\mu }^{T}d$ to be nonnegative. This proves the strict separation of disjoint linear polyhedra using Lagrangian duality.

Choose the oriented incidence matrix convention with $+1$ at an edge's tail and $-1$ at its head. Let $b_i$ be net supply, so ${\sum }_i{b}_i=0$. The uncapacitated minimum-cost flow problem on a finite directed graph isor $Bf=b$. Negative $b_i$ denotes net demand. Costs may have either sign; the problem can be infeasible or unbounded.

For vertex network dual potentials $\pi$, the optimization Lagrangian isThe infimum over $f\ge 0$ is finite exactly when every network reduced cost $r_{ij}=c_{ij}-{\pi }_i+{\pi }_j$ is nonnegative. Thus the Lagrangian dual problem and complementary slackness areA feasible flow and feasible network dual potentials satisfying these equalities have equal costs and are optimal by weak duality.

If the underlying undirected graph is connected, deleting one redundant balance row makes the oriented incidence matrix have rank $|V|-1$. A set of $|V|-1$ independent edge columns is exactly a spanning tree. Set all non-tree flows to zero, solve the tree balances, and check nonnegativity to obtain a basic feasible solution. Basic tree edges may have zero flow, which is degeneracy in linear programming. Requiring $r_{ij}=0$ on tree edges determines network dual potentials up to a common additive constant. If every non-tree network reduced cost is nonnegative, the tree flow is optimal. This is the basis of the network simplex algorithm. If the graph is disconnected, solve the balances separately in each component; each component must have total net supply zero and uses its own spanning tree.

In congestion control, resource $j$ observes only its own aggregate load $y_j={\sum }_{r:j\in r}{x}_r$ and generates the nonnegative feedback price ${\mu }_j=p_j(y_j)$. User $r$ receives the sum $M_r={\sum }_{j\in r}{\mu }_j$ of prices along its route. Its desired increase $w_r$ competes with the congestion charge $x_r{M}_r$, while ${\kappa }_r$ sets the adjustment speed. This is local information: a resource needs the rates passing through it, and a user needs the feedback along its own route, rather than the complete network state.

There is a hypothesis to make explicit before claiming global convergence. If “increasing” means nondecreasing, the printed assumptions allow $p_j\equiv 0$, in which case ${\dot{x}}_r={\kappa }_r{w}_r>0$ and there is no equilibrium point. The intended theorem holds when every route contains at least one resource whose price is not identically zero, with nonempty routes and nonnegative, continuous, nondecreasing prices. Strictly increasing prices satisfy this condition automatically. We prove the theorem under this necessary qualification; without it the zero-price example disproves the literal claim.

Let $P_j(y)={\int }_0^y{p}_j(u)du$. Each $P_j$ is a nonnegative convex function. For positive rates define the primal congestion potentialThe weighted logarithmic utility is a strictly concave function, and subtracting the resource costs preserves strict concavity. Thus $U$ has at most one maximizer and at most one critical point on the positive orthant.

It also attains its maximum there. A nonzero nondecreasing price is bounded below by some positive constant at all sufficiently large loads; hence its primitive satisfies $P_j(y)\ge c_{j}y-K_j$ with $c_j>0$. Choose $c_j=0$ at zero-price resources. The route coverage assumption gives $c={\min }_r{\sum }_{j\in r}{c}_j>0$, soThis bounds all superlevel sets of $U$. Within a bounded set, approaching $x_r=0$ makes $U\to -\infty$. Consequently every nonempty superlevel set is a compact set contained in the positive orthant. A maximizing sequence stays in one such set, giving a unique interior maximizer $x^{\ast }$. Its critical-point equations areThe rate dynamics are the gradient flow with diagonal mobility ${\dot{x}}_r={\kappa }_r{x}_r{g}_r(x)$. Along any positive trajectory,with equality only at $x^{\ast }$. Thus $V(x)=U(x^{\ast })-U(x)$ is a strict Lyapunov function. The initial superlevel set traps the trajectory away from both infinity and the boundary, ensuring global continuation. Positive initial rates remain positive: variation of constants in ${\dot{x}}_r+{\kappa }_r{M}_r(t)x_r={\kappa }_r{w}_r$ makes this explicit. Starting with finite nonnegative rates also gives positive rates at every later positive time.

For completeness, mere continuity of $p_j$ suffices for convergence; one need not silently assume differentiable prices. On the trapping compact set, both the vector field and $D(x)={\sum }_r{\kappa }_r{x}_r{g}_r(x{)}^2$ are continuous and bounded. The trajectory satisfies a uniform Lipschitz condition in time, so $D(x(t))$ is uniformly continuous. Also ${\int }_0^{\infty }D(x(t))dt<\infty$, since $U$ increases to a finite limit. The decay of a nonnegative uniformly continuous integrable function applies: otherwise disjoint intervals of a fixed positive width would each contribute a fixed positive amount. Every accumulation point therefore has $D=0$ and is $x^{\ast }$. Compactness then proves the global convergenceThis is the dissipation argument behind the LaSalle invariance principle. If uniqueness of trajectories is desired under continuous prices, put $z_r=2\sqrt{x_r/{\kappa }_r}$. The transformed equation is $\dot{z}={\nabla }_{z}U(\kappa z^2/4)$. The transformed potential is concave: it is a sum of positive multiples of $\log z_r$, minus convex nondecreasing $P_j$ composed with nonnegative quadratic sums. Its gradient is monotone decreasing, so the squared distance between two solutions cannot increase. This proves uniqueness for positive initial data without a locally Lipschitz function hypothesis on $p_j$.

For the alternative price dynamics, restrict first to positive prices on resources used by at least one route, and discard unused resources. Define the dual congestion potentialIts domain has ${\mu }_j\ge 0$ and $M_r>0$ for every route. Strictly increasing $q_j$ make each $Q_j$ strictly convex, and $-\log M_r$ is convex. Thus $H$ is a strictly convex function. Each $q_j$ is eventually bounded below by a positive constant, so its integral grows at least linearly, dominating the negative logarithms as $\Vert \mu \Vert \to \infty$. If a route price tends to zero in a bounded set, $H\to \infty$. Therefore $H$ attains a unique minimum on its domain including the admissible boundary faces.

That minimum has every used-resource price positive. At a face ${\mu }_j=0$, all route prices are still positive and the inward derivative isHence the minimum cannot lie on that face. At the unique interior minimum,Since the price dynamics are ${\dot{\mu }}_j=-{\kappa }_j{\mu }_j{\partial }_{j}H$, their positive equilibrium points are exactly these critical points. There is one equilibrium in the positive price domain. An unused resource instead has its only nonnegative equilibrium at zero.

The positive-domain qualification matters here even with strictly increasing $q_j$. If zero prices are admitted, the multiplicative factor ${\mu }_j$ permits boundary equilibria of multiplicative price dynamics. Take one route using two resources, $w=1$, and $q_1(a)=q_2(a)=a$. Both $({\mu }_1,{\mu }_2)=(1,0)$ and $(0,1)$ are equilibria, with $x=1$, as is the positive equilibrium $(1/\sqrt{2},1/\sqrt{2})$, with $x=1/\sqrt{2}$. Thus the literal assertion of uniqueness on the whole nonnegative orthant is false. A boundary equilibrium need not satisfy load equals supplied capacity at its zero-price resources.

The relationship between the two intended equilibria is inverse resource response. The first system has ${\mu }_j=p_j(y_j)$; the positive equilibrium of the second has $y_j=q_j({\mu }_j)$. If $q_j=p_j^{-1}$ on the relevant ranges, these are the same resource equations, and both systems also impose $x_r{M}_r=w_r$. They therefore have the same rates and prices. For strictly increasing continuous $p_j$ with $p_j(0)=0$ and unbounded range, the ordinary inverses satisfy all the stated conditions on $q_j$. With arbitrary independently chosen $p_j$ and $q_j$, the equilibria need not agree. Under inverse response, $P_j$ and $Q_j$ are convex conjugates on the nonnegative half-line, and $H$ supplies the Lagrangian dual problem for maximizing the weighted logarithmic utility minus resource costs, up to constants independent of $\mu$.

For the same multiplier convention, the Lagrangian dual problem isFor every feasible $x$ and every $\lambda$, the equality constraint implies $q(\lambda )\le L(x,\lambda )=f(x)$. Taking first the infimum over feasible $x$ and then the supremum over $\lambda$ proves weak duality:The inequalities remain meaningful with the usual extended-value conventions, including $q(\lambda )=-\infty$ when the Lagrangian is unbounded below. Equality of optimum values, or their attainment, does not follow from weak duality. If the sufficient pair in part (a) exists, it attains equality, since $q({\lambda }^{\ast })=L(x^{\ast },{\lambda }^{\ast })=f(x^{\ast })$.

Associate nonnegative Lagrange multipliers $\alpha$ with $-Pz-1\le 0$ and $\beta$ with $Pz-1\le 0$. The Lagrangian dual problem is obtained fromIts infimum over unrestricted $z$ is finite exactly when $P^T(\alpha -\beta )=x$. Thus the dual isThe precise finiteness condition is $x\in \mathrm{range}{P}^T$. If $x=P^{T}q$, then $x^{T}z=q^{T}Pz\ge -\Vert q{\Vert }_1$ on the feasible set. The primal is feasible and bounded below, and splitting $q$ into positive and negative parts makes the dual feasible. Linear programming duality therefore gives attained equal finite optima. Equivalently, the strictly feasible point $z=0$ satisfies the Slater condition, with the required finiteness hypothesis. This is strong duality in the usual finite sense.

If $x\notin \mathrm{range}{P}^T=(\ker P{)}^{\perp }$, there is $d\in \ker P$ with $x^{T}d<0$. Every $z=td$, $t\ge 0$, is feasible and the objective tends to $-\infty$. The dual is then infeasible. Equality of extended values still holds if its supremum over the empty feasible set is defined as $-\infty$, but there are no finite optimizers. The paper does not assume full column rank of $P$, so this qualification is necessary. The same calculation gives the support function of an inverse image of an infinity-norm ball, ${\inf }_r{r}^{T}x=r_0^{T}x-{\min }_{P^{T}q=x}\Vert q{\Vert }_1$ for finite decisions.

For the equality constraint, the Lagrangian and dual function areand the Lagrangian dual problem is ${\sup }_{z\in {\mathbb{R}}^m}q(z)$. Weak duality says $q(z)\le p^{\ast }$ for every $z$. Strong duality means the dual supremum equals the primal infimum, usually with a dual maximizer. A sufficient convex constraint qualification is that $f$ be proper, closed and convex and that some $x\in \mathrm{ri}(\mathrm{dom}f)$ satisfy $Ax=b$.

Use the sign conventionfor the Lagrangian function in constrained optimization. The Lagrangian dual problem iswhere $f^{\ast }$ is the convex conjugate. For this convex problem with affine equality constraints, the stationarity and feasibility parts of the Karush-Kuhn-Tucker conditions areThey say exactly that the displayed operator satisfiesThus its zeros are precisely the primal-dual optimal points, subject to the usual attainment assumptions.

For $u=(x,z)$ and $\tilde{u}=(y,s)$, the Euclidean inner product givesThe last two terms cancel by the defining property of the matrix transpose, and the first is nonnegative by part a. Hence $F$ is a monotone operator.

Associate a nonnegative Lagrange multiplier ${\lambda }_i$ with each inequality. The Lagrangian dual problem begins withIts infimum over $x\in {\mathbb{R}}^n$ is finite exactly when $A^T\lambda +c=0$, in which case it equals $-b^T\lambda$. The dual linear program is consequentlyFor any primal-feasible $x$ and dual-feasible $\lambda$,which proves weak duality. Strong duality means equality of the two optimal values. The stated strict feasibility is the Slater condition; together with finiteness of the primal optimum it gives strong duality and an attained dual optimum ${\lambda }^{\ast }$.

A primal-dual optimal point consists of a feasible solution of an optimization problem and a feasible solution of its Lagrangian dual problem whose objective values agree. Under differentiability and suitable convexity, such points satisfy the Karush-Kuhn-Tucker conditions.

The original mathematical optimization problem relative to a chosen Lagrangian dual problem. In convex perturbation duality, its objective is $\phi (x)=f(x,0)$ and its value is $p(0)$. Strong duality concerns optimal values; primal attainment additionally requires an actual minimizing point.

Nonempty disjoint linear polyhedra $P=\{x:Ax\le b\}$ and $Q=\{y:Cy\le d\}$ admit $h$ with a strictly positive separation gap. A phase-I linear program minimizing common constraint violation has positive attained optimum. Its Lagrangian dual problem gives nonnegative $\lambda ,\mu$ with $A^T\lambda +C^T\mu =0$ and ${\lambda }^{T}b+{\mu }^{T}d<0$. Then $h=A^T\lambda$ satisfies $h^{T}x\le {\lambda }^{T}b<-{\mu }^{T}d\le h^{T}y$. The polyhedral hypothesis matters; arbitrary disjoint closed convex sets need not have a positive separation gap.