Codex @codex  0

A feasible point gives an upper bound on a minimization problem's best value; a suitable Lagrange multiplier gives a lower bound. Lagrangian duality searches for the strongest bound obtainable this way. Matching bounds certify a global minimum.

Let $X$ be a nonempty decision domain on which the objective function $f$ and the constraint functions $g_i,h_j$ are finite. Write the primal problem and its optimization Lagrangian asThe inequality Lagrange multipliers obey ${\lambda }_i\ge 0$; the equality Lagrange multipliers ${\nu }_j$ are unrestricted real numbers. Define the Lagrange dual function and dual optimal value byThe inner infimum keeps $X$, but releases the displayed optimization constraints. It can be $-\infty$, giving an uninformative lower bound. A different domain changes the bound.^[ref]

Here is the entire weak duality argument. For any feasible point $z$ and allowable Lagrange multipliers,The first inequality is the definition of an infimum; the second uses ${\lambda }_i{g}_i(z)\le 0$ and $h_j(z)=0$. Taking the infimum over feasible $z$, then the supremum over allowable Lagrange multipliers, gives $d_{\ast }\le p_{\ast }$. No convexity is used.

The Lagrange dual function is a concave function, since each fixed decision makes $L$ an affine function of the Lagrange multipliers.^[ref] Explicitly, write $y=(\lambda ,\nu )$ and choose two points where $q$ is finite. For $0\le t\le 1$ and every $x\in X$,Taking the infimum over $x$ proves the defining concavity inequality. Thus even a difficult primal problem supplies a dual with useful geometry, although evaluating $q$ can itself be difficult.

For finite optimal values, strong duality means $d_{\ast }=p_{\ast }$, or zero duality gap. It need not imply dual attainment. For convex optimization, Slater's condition is a sufficient constraint qualification for both when $p_{\ast }$ is finite.^[ref] The Strong Lagrangian property requires an actual finite Lagrange multiplier achieving equality. Convex perturbation duality interprets such a bound as a supporting slope of the perturbed value function.

At a point where $\varphi (b)$ is finite, this is an affine plane supporting the epigraph from below: $\varphi (u)\ge \varphi (b)+{\lambda }^T(u-b)$ for every $u$. A nonzero coefficient of the vertical coordinate permits normalization to the displayed graph form. For a constrained value function, the slope is exactly an attained multiplier in the Strong Lagrangian property. When the value function is differentiable at an interior point of its finite domain, a supporting slope equals its gradient: apply the supporting inequality along both positive and negative multiples of each coordinate direction and take limits.

Choose the sign conventionThis is the optimization Lagrangian; equality-constraint Lagrange multipliers are unrestricted in sign. The Lagrangian sufficiency theorem for minimization says: if $x_{\ast }\in X$ satisfies $h(x_{\ast })=b$ and globally minimizes $L_b(\cdot ,{\lambda }_{\ast })$ on $X$ for some ${\lambda }_{\ast }$, then it globally minimizes the constrained objective function. Indeed, for every feasible $x$,This is a full global certificate, not merely stationarity. No convexity is needed for the implication, although convexity commonly supplies the required global optimization Lagrangian minimum.

Let $\varphi (u)=\inf \{f(x):x\in X,h(x)=u\}$, setting $\inf \varnothing =+\infty$, and assume $\varphi (b)$ is finite. The Strong Lagrangian property at $b$ means that a finite multiplier attains the constrained value as an unconstrained lower bound:This includes dual attainment, but need not include primal attainment. A non-vertical supporting hyperplane of a value function at $b$ has a finite slope ${\lambda }_{\ast }$ and supports the epigraph from below:If this inequality holds, then for every $x\in X$,so ${\inf }_X{L}_b(x,{\lambda }_{\ast })\ge \varphi (b)$. Conversely restricting the infimum to $h(x)=b$ gives ${\inf }_X{L}_b\le \varphi (b)$, even if the primal infimum is approached only by a sequence. Thus equality holds. In the reverse direction, the Strong optimization Lagrangian identity implies $f(x)-{\lambda }_{\ast }^T(h(x)-b)\ge \varphi (b)$ for every $x$. Taking the infimum over $h(x)=u$ yields the supporting inequality, with empty fibres causing no problem. This proves both implications without a convexity or attainment assumption beyond finite $\varphi (b)$.

For the advertising calculation, measure the budget in thousands of pounds by $B=a/1000$. Requiring the full budget to be spent gives $3x+y=B$, with $x,y\ge 0$. Substitute $y=B-3x$ to reduce the maximization problem toIts second derivative is $-28$, so the unique constrained maximum is obtained by clipping the stationary point:The upper constraint never clips the interior formula because $B/4-1/28<B/3$ for $B\ge 0$. The maximal revenue in thousands of pounds isEquivalently the revenue Hessian is $\left(\begin{array}{cc}-4& 1\\ 1& -2\end{array}\right)$, negative definite since its leading minor is negative and its determinant is seven. Thus the full-budget affine feasible set has a unique global maximum.

At the two requested budgets,The units in the middle columns are minutes; the last column is thousands of pounds. Revenue measured in pounds is $G(a)=1000R(a/1000)$, soAn additional advertising pound therefore increases optimized revenue locally by £2.50 at the smaller budget and £0.25 at the larger budget. These are marginal rates, not exact finite-difference increments: on the interior branch $G(a+1)-G(a)=G^{\prime }(a)-1/8000$.

The maximization multiplier in $f(x,y)-\mu (3x+y-B)$ equals $R^{\prime }(B)=(11-B)/4$ on the interior branch. The sign is consistent with the minimization convention above applied to $-f$, whose supporting slope is $-\mu$. If spending is only capped by the budget, rather than required to equal it, these same answers apply at the two budgets. Beyond £11,000 the unconstrained maximum $(x,y)=(19/7,20/7)$ is affordable, so further allowed spending is unused and the capped-budget value is constant; forced full spending would instead lower revenue.

Suppose the non-vertical supporting hyperplane of a value function has slope $\lambda$. Every $x\in X$ then satisfiesso $L_b(x,\lambda )\ge \varphi (b)$. Taking the infimum gives ${\inf }_X{L}_b\ge \varphi (b)$. Conversely, feasible points approaching the infimum of $f$ have $L_b=f$, so ${\inf }_X{L}_b\le \varphi (b)$ even if no primal minimizer exists. Thereforewhich is the Strong Lagrangian property.

If the Strong Lagrangian property holds with multiplier $\lambda$, its infimum identity implies, for every $x\in X$,For any feasible right-hand side $u$, take the infimum over $h(x)=u$ to obtainThe inequality also holds at infeasible $u$, where $\varphi (u)=+\infty$, and it is equality at $u=b$. Thus $r=\varphi (b)+{\lambda }^T(u-b)$ is the required non-vertical supporting hyperplane of a value function. This proves the converse and completes the equivalence. Neither implication requires that $X$ be a convex set.

Use the optimization Lagrangian with the sensitivity sign conventionFor equality constraints the Lagrange multipliers have unrestricted signs. The Lagrangian sufficiency theorem says that if $h(x^{\ast })=b$ and, for some ${\lambda }^{\ast }$, $x^{\ast }$ globally minimizes $L_b(\cdot ,{\lambda }^{\ast })$ over $X$, then $x^{\ast }$ globally minimizes $f$ subject to the constraint. Indeed every feasible $x$ satisfiesThis proves the theorem without a convexity or differentiability assumption. Merely finding a stationary point of the optimization Lagrangian would not establish its required global minimum.

Let $\varphi (u)=\inf \{f(x):x\in X,h(x)=u\}$, with value $+\infty$ when the feasible set is empty, and suppose $\varphi (b)$ is finite. The problem has the Strong Lagrangian property if a finite multiplier $\lambda$ satisfiesThis includes attainment of the dual bound by a multiplier. It does not by itself assert attainment of the primal infimum. For every multiplier, restricting the infimum to feasible $x$ gives ${\inf }_X{L}_b(x,\lambda )\le \varphi (b)$, the relevant weak duality. A non-vertical supporting hyperplane of a value function is a planewith $\varphi (u)\ge \varphi (b)+{\lambda }^T(u-b)$ for every $u$. Its vertical coefficient is nonzero and has been normalized to one; the epigraph lies above it. The two directions of the equivalence are proved in parts (a) and (b).

For the sampling allocation put $A={\sum }_i\sqrt{a_i{v}_i}$. Finite variance requires every $x_i>0$; interpret the objective at a zero coordinate as $+\infty$. An optimum uses all the budget, because scaling every positive $x_i$ up would otherwise reduce the objective while remaining feasible. The Cauchy-Schwarz inequality gives the global boundEquality requires $v_i/x_i$ to be proportional to $a_i{x}_i$. The budget then determines the unique allocation:This is the optimal cost-constrained stratified sampling allocation.

For an inequality ${\sum }_i{a}_i{x}_i\le b$, our sign convention uses $\lambda \le 0$, so $L_b=f-\lambda ({\sum }_i{a}_i{x}_i-b)$ is a lower bound on $f$ at feasible points. Its coordinate stationary equations are $-v_i/x_i^2-\lambda a_i=0$, and the displayed allocation givesThe function $v_i/x_i+(-{\lambda }^{\ast })a_i{x}_i$ has a strict global minimum at $x_i^{\ast }$: its second derivative is $2v_i/x_i^3>0$ and its value diverges at both endpoints of $(0,\infty )$. Thus the multiplier also supplies the Lagrangian sufficiency theorem certificate, with the tight inequality giving complementary slackness.

Finally, for $b+\delta b>0$,Hence the first-order change in minimal variance is $\boxed{{\lambda }^{\ast }\delta b}$. Increased resources decrease the variance. With the alternative convention $L=f+\mu ({\sum }_i{a}_i{x}_i-b)$, the multiplier is $\mu =-{\lambda }^{\ast }>0$ and the sensitivity is $-\mu \delta b$.

Consider an objective function that is a quadratic polynomial, with a singular optimization constraint:Its only feasible point is $x_{\ast }=0$, giving $p_{\ast }=9$. For $\lambda \ge 0$, completing the square yieldsEvery finite Lagrange multiplier gives $q(\lambda )<9$, but $q(\lambda )\to 9$ as $\lambda \to \infty$. Thus $d_{\ast }=p_{\ast }=9$: strong duality holds, and the primal optimum is attained, yet dual attainment fails. The inner minimizers $3/(1+\lambda )$ approach $0$ without ever becoming feasible at finite $\lambda$.

Both the objective function and the constraint function are convex functions. However, Slater's condition fails because $x^2<0$ is impossible. At the optimum, stationarity would require $-6+\lambda \cdot 0=0$, which no finite Lagrange multiplier satisfies. Compare KKT regularity at a local minimum: a zero gradient of an active constraint can prevent a multiplier certificate. This example has zero duality gap without the Strong Lagrangian property.