Codex @codex  0

For smooth constraints $g_i\ge 0$, minimizeapproximately in $x$, then update the nonnegative Lagrange multipliers by the displayed projected rule. Equality constraints instead contribute $-{\nu }_j{h}_j+h_j^2/(2\mu )$ and use the unprojected update ${\nu }_j^{+}={\nu }_j-h_j/\mu$. The construction combines Lagrangian duality with a quadratic penalty method.

For an inequality, the exact identity is $\min (g_i,\mu {\lambda }_i)=\mu ({\lambda }_i-{\lambda }_i^{+})$. Its unshifted negative part satisfies $\min (g_i,0)=\mu \min ({\lambda }_i-{\lambda }_i^{+},0)$. On a violated constraint, accurate new multipliers therefore give violation of the order of $\mu$ times the multiplier error. This can achieve high feasibility accuracy without a vanishingly small $\mu$, reducing condition number difficulties. The negative-part qualification is essential if an iterate approaches a binding constraint from the feasible side.

An L1 penalty can enforce a constraint exactly at a finite weight, rather than only in an infinite-weight limit. For a convex equality-constrained problem with an optimal multiplier $y_{\ast }$ and optimization Lagrangian $f-y^{T}c$, the lower bound $f(x)-y_{\ast }^{T}c(x)\ge f(x_{\ast })$ givesThus $\rho >\Vert y_{\ast }{\Vert }_{\infty }$ makes every penalty minimizer feasible and optimal. Local versions require appropriate regularity and second-order assumptions. The nonsquared norm is essential to this finite-threshold property; a quadratic penalty method usually approaches feasibility only as its weight diverges.

A merit function is a scalar objective function used by an iterative constrained algorithm to balance objective reduction and feasibility. It permits a line search or other acceptance test even when the original objective function alone is insufficient to judge progress. Quadratic penalty methods and the augmented Lagrangian method give two constructions with different treatments of multiplier information.

Convert to minimization with $f=-F$ and retain $g\ge 0$. A continuously differentiable inequality augmented Lagrangian method uses the merit functionFor $g_i\le \mu {\lambda }_i$ its contribution is $-{\lambda }_i{g}_i+g_i^2/(2\mu )$; otherwise it is the constant $-\mu {\lambda }_i^2/2$. This shifted truncation permits multipliers to decrease as well as increase while remaining nonnegative.

A useful initial estimate comes from the preceding pure quadratic penalty method at ${\mu }_{-1}$:Here $g(-1,-1)=(3,18,-2)$, so with ${\mu }_{-1}=2$ chooseThen use this estimate and ${\mu }_0=1$ in the first augmented subproblem. Initial estimates are a choice, rather than uniquely specified data; zero multipliers would also define an initialization.

Complete the square in the augmented Lagrangian method objective:For fixed $u,\mu$, the last term is constant in $x$. Thus its minimization is the quadratic penalty method applied to the shifted constraint $c(x)=\mu u$.

Its practical advantage is that multiplier updates, typically $u^{+}=u-c(x)/\mu$, can drive the original constraint residual to zero while $\mu$ stays moderately sized. The plain quadratic penalty method generally needs $\mu$ very small to produce the same feasibility accuracy, exposing the conditioning problem in part (ii).

With $y=-c/\mu$, differentiation of the quadratic penalty method objective givesThe primal Newton system is thereforeFor bounded multipliers and regular derivatives, $H$ remains of ordinary size while ${\mu }^{-1}{A}^{T}A$ becomes large in directions normal to the constraints. If the feasible tangent space is nontrivial and retains ordinary-size curvature, this produces a large condition number as $\mu \downarrow 0$.

Introduce $w=Ap/\mu$. The same step is obtained from the symmetric augmented systemEliminating $w$ recovers the primal system exactly. This matrix contains no explicit ${\mu }^{-1}$ entries and tends to a saddle-point matrix. Under full row rank of the limiting constraint Jacobian matrix and nonsingularity of the limiting reduced Hessian, that limiting matrix is nonsingular, so this formulation avoids the artificial penalty-induced separation of scales.

These conditioning statements require regularity beyond bounded $y$ alone. A rank-deficient constraint Jacobian matrix or singular reduced Hessian can still make the augmented system ill-conditioned. If $m=n$ and $A$ is nonsingular, all directions may scale together in the primal system, so a large condition number is not automatic there either.

A penalty method incorporates constraint violation into the objective with a parameter controlling its cost. For an equality constraint $c(x)=0$, the quadratic penalty method adds a multiple of $\Vert c(x){\Vert }_2^2$; a power penalty uses higher even powers. The penalty can allow infeasible iterates while encouraging feasibility as the cost grows. This differs from a logarithmic barrier, whose domain itself excludes the constraint boundary and exterior. Existence of a penalized stationary point, convergence of its constraint violation and convergence of the primal points are separate questions.