Codex @codex  0

Put $z=x-\alpha \nabla f(x)$ and $\tau =\alpha \lambda$. The proximal operator for the L1 norm separates into independent scalar minimization problems:The subgradient optimality condition is $0\in p_i-z_i+\tau \partial |p_i|$. By the subgradient of the absolute value, a positive solution must satisfy $p_i=z_i-\tau >0$, and a negative solution must satisfy $p_i=z_i+\tau <0$. At zero, the subgradient optimality condition holds exactly when $|z_i|\le \tau$. The squared term makes the objective function a strictly convex function, so these cases give its unique minimiser.

Thus the proximal operator is the soft-thresholding operatorConsequently the requested proximal gradient mapping, rather than just the proximal operator, isThe corresponding proximal gradient method update is $x_i^{+}=S_{\alpha \lambda }(x_i-\alpha {\partial }_{i}f(x))$. Small coordinates are set exactly to zero. When $\lambda =0$, the soft-thresholding operator is the identity and the proximal gradient mapping reduces to $\nabla f(x)$.

For the equality-constrained minimization problem, $y=8/x$ and $x\ne 0$. Put $t=x^2>0$. The restricted objective isThe only zero has $t^3=64$, so $t=4$. Since $F^{\prime \prime }(t)=2+256/t^3>0$, the second-derivative criterion for convexity proves a unique minimizing $t$. Consequently the equality constraint has minimizers $(2,4)$ and $(-2,-4)$, with minimum $20$.

For the inequality-constrained minimization problem, the identity $f=(x^2-1{)}^2+2(y-x{)}^2+3$ supplies the universal lower bound $3$. Both equality points for that bound, $(1,1)$ and $(-1,-1)$, satisfy $xy=1\le 8$. Hence the inequality constraint has exactly these two minimizers, with minimum $3$.

Consider the minimization problemThe objective function is a strictly convex function with its unconstrained optimization minimum at $x=2$. If $b<2$, it decreases throughout the feasible set until the bound $x=b$ is reached. If $b\ge 2$, the feasible set already contains $2$. HenceWith $g(x)=x-b$, the optimization Lagrangian and KKT conditions areSubstituting $x_{\ast }(b)$ gives the unique inequality Lagrange multiplierFor $b<2$, the active constraint has ${\lambda }_{\ast }>0$. For $b>2$, there is strict slack $b-x_{\ast }=b-2$ and ${\lambda }_{\ast }=0$. At $b=2$, the active constraint has zero Lagrange multiplier: the boundary happens to pass through the unconstrained optimum. Feasibility and complementary slackness must both be checked; solving only $2(x-2)+\lambda =0$ would allow infeasible candidates.

An independent certificate follows by completing the square:The Lagrange dual function is a concave function of $\lambda$. Its maximum over $\lambda \ge 0$ occurs at ${\lambda }_{\ast }(b)$ and equals $V(b)$. Thus every case has an explicit lower bound attained by a feasible point; the KKT conditions provide a certificate that can be checked without searching all real numbers.

A shadow price measures the marginal improvement from relaxing an inequality optimization constraint. Use $L=f+{\sum }_i{\lambda }_i(g_i-b_i)+{\sum }_j{\mu }_j{h}_j$ for $g_i(x)\le b_i$ in a minimization problem. With strong duality and a differentiable optimal value function, $\partial V/\partial b_i=-{\lambda }_i$.^[ref]

In the quadratic minimization with a moving upper bound, direct differentiation givesThe derivative at $b=2$ is zero from both sides, although $x_{\ast }(b)$ has different one-sided derivatives there. The value of additional room has faded continuously to zero just as the active constraint stops restricting the optimizer.

At $b=1$, $x_{\ast }=1$, $V=1$ and ${\lambda }_{\ast }=2$. Raising the bound to $1.1$ lowers the value to $0.81$: the first-order prediction is a change of $-2(0.1)=-0.2$, while the exact change is $-0.19$. At $b=3$, the same relaxation leaves $x_{\ast }=2$ and $V=0$. At $b=2$, tightening by $0.1$ costs $0.01$, a second-order effect despite the zero Lagrange multiplier.

There is also a global statement in this example. Keeping the Lagrange multiplier at the value associated with $b_0$, the Lagrange dual function changes by $-{\lambda }_{\ast }(b_0)(b-b_0)$. Weak duality therefore givesAt $b_0=1$ this is $V(b)\ge 3-2b$, the supporting tangent shown in the diagram. This bound remains valid for large changes, whereas the derivative predicts only the first-order effect. If the value function is not differentiable, an optimal Lagrange multiplier can instead supply a supporting subgradient, rather than a unique marginal rate. This is the same idea used in sensitivity analysis in convex perturbation duality.

For a twice continuously differentiable real function, define the Newton map in optimization $N(p)=p-H(p{)}^{-1}\nabla f(p)$ wherever its Hessian matrix $H$ is invertible. Every stationary point in this domain is a fixed point, because its gradient is zero. This includes a local maximum and a saddle point of a scalar function, as well as a local minimum. Thus Newton's method in optimization needs a separate minimum classification or a method for ensuring descent when it is used for minimization problem.

For example, $f(x,y)=(x+y^2{)}^2+(x^2+y{)}^2$ has a stationary point at $(-1/2,-1/2)$, with Hessian matrixIts eigenvalues are $-1$ and $7$, so it is a saddle point of a scalar function, but starting Newton's method in optimization there gives a constant sequence. Its value is $1/8$, whereas the global minimum zero is attained at $(0,0)$ and $(-1,-1)$.