Codex @codex  0

The constraint implies $x^2=z^2+1$, so $x\ne 0$ and its gradient $(2x,0,-2z)$ never vanishes on the feasible set. The Lagrange multiplier equations $\nabla f=\lambda \nabla g$ giveThus $\lambda =1$, $y=0$, $z=-1/2$, and feasibility gives $x=\pm \sqrt{5}/2$. The two stationary points and their values areThey are both global minima. Indeed, on the constraint, completing the square givesEquality holds only at the two displayed points. There are no other constrained stationary points, and the objective is unbounded above as $|y|\to \infty$.

Put $h=x-1$ and $k=y-1$. At $(1,1)$, the value is $5+4\alpha$, the gradient is $(5+4\alpha ,4+2\alpha )$, and the Hessian matrix is $\left(\begin{array}{cc}8& 2\alpha \\ 2\alpha & 2\end{array}\right)$. Thus the quadratic Taylor expansion isThe exact remainder is $h^3$, since the original polynomial has only the cubic term $x^3$ beyond degree two.

Let $g_1=x^2+y^2+z^2-4$ and $g_2=(x-3{)}^2+y^2+z^2-4$. Subtracting the two constraint equations gives $-6x+9=0$, hence $x=3/2$ and $y^2+z^2=7/4$. The feasible set is therefore a circle.

For Lagrange multipliers $\lambda ,\mu$, the stationarity equation $\nabla f=\lambda \nabla g_1+\mu \nabla g_2$ becomesAt $x=3/2$, the first equation gives $\lambda =\mu$. The second ensures $\lambda +\mu \ne 0$ and $y\ne 0$, so the third gives $z=0$. The constraints then force $y=\pm \sqrt{7}/2$. Both points do satisfy the multiplier equations, taking $\lambda =\mu =1/(4y)$.

There are no missing singular constraint points: on the feasible circle, the two constraint gradients $(3,2y,2z)$ and $(-3,2y,2z)$ are linearly independent, since $y^2+z^2=7/4>0$. Thus the complete list of constrained critical points and critical values isThe bound $|y|\le \sqrt{7}/2$ on the feasible circle shows that the positive value is the global maximum and the negative value is the global minimum.

The gradient vanishes when $4x+3y-1=0$ and $3x+2y-1=0$. Subtracting these equations gives $x+y=0$, so the unique critical point is $\boxed{(x,y)=(1,-1)}$, with $f(1,-1)=0$.

The Hessian matrix isIt is indefinite, so the critical point is a saddle point. Explicitly, around that point the change is exactly $2u^2+3uv+v^2$: it is positive for $v=0,u\ne 0$ and negative for $v=-3u/2,u\ne 0$. Thus arbitrarily close points have both larger and smaller values.

On the unit circle the constraint gradient $(2x,2y)$ never vanishes. The Lagrange multiplier equations areA nonzero solution of the first two equations requiresso $\lambda =9/2$ or $\lambda =-1/2$. Multiplying the first equation by $x$, the second by $y$, and adding shows $2f=2\lambda (x^2+y^2)=2\lambda$. Thus each candidate value of $f$ equals its multiplier.

For $\lambda =9/2$, $y=-x/3$, giving $(x,y)=\pm (3,-1)/\sqrt{10}$. For $\lambda =-1/2$, $y=3x$, giving $(x,y)=\pm (1,3)/\sqrt{10}$. Since a continuous function attains its extrema on the compact set given by the unit circle and these exhaust the Lagrange multiplier candidates,Each $\pm$ changes both coordinates together.

The gradient equations areThus $x=3$ gives $y=2$, while $x=-3$ gives $y=8$. The Hessian matrix isAt both critical points the determinant is $-36<0$, so the Hessian is indefinite and the second-derivative test classifies both as saddle points. Direct substitution givesThe factorization $f+30=(x^2-9)(x+y-5)$ also shows the crossing level-set branches through each saddle.

The gradient vanishes exactly when $x^2/4-1=0$ and $y^2/4-1=0$, giving the four points $(\pm 2,\pm 2)$. The Hessian matrix is $\mathrm{diag}(x/2,y/2)$. It is positive definite at $(2,2)$, negative definite at $(-2,-2)$, and indefinite at the two mixed-sign points. Thus $(2,2)$ is a strict local minimum, $(-2,-2)$ a strict local maximum, and $(2,-2),(-2,2)$ are saddle points by the second-derivative test. Their respective function values are $-8/3$, $8/3$, and zero at both saddle points.

The continuous function attains a global maximum and minimum on the closed disc by the extreme value theorem. All four unconstrained stationary points have squared distance eight, so none lies in its interior. The extrema must therefore lie on $x^2+y^2=6$.

The constraint gradient is nonzero on that circle, so Lagrange multipliers apply:Subtracting gives $(x-y)((x+y)/4-2\lambda )=0$. If $x=y$, the constraint gives $x=y=\pm \sqrt{3}$. Otherwise $2\lambda =(x+y)/4$; substitution into either equation gives $xy=-4$. This is impossible, since $|xy|\le (x^2+y^2)/2=3$ on the circle. Thus the diagonal points are the only candidates. Comparing their values gives

Use the LagrangianThe Lagrange multiplier equations areFeasibility makes $x\ne 0$, so $\lambda =2x^2>0$. The second equation becomes $2y(8y^2+\lambda )=0$, forcing $y=0$ and then $x=\pm 1$. The constraint gradient $(2x,-2y)$ is nonzero on the feasible set, so the multiplier calculation has no missing singular feasible point.

To verify the global minimum rather than just stationarity, eliminate $x^2=1+y^2$:Equality holds exactly at $(x,y)=(1,0)$ or $(-1,0)$, with minimum value $1$.

At $d=1000$, gradient descent and full BFGS are practical candidates; a dense Newton method can also be feasible, but its extra work and indefinite curvature need attention. The number of examples $n$, data sparsity and conditioning still matter.

For dense $A$, a gradient evaluation costs $O(nd)$ arithmetic, with $O(d)$ working storage beyond the data and accumulation buffers. Gradient descent is inexpensive per iteration and, with an appropriate line search, follows a descent direction, but poor conditioning can make it slow.

Forming the Hessian matrix explicitly costs $O(n{d}^2)$ arithmetic, and a dense Newton method solve costs $O(d^3)$. A $1000$-square double-precision matrix uses about $8$ MB, so storage alone is manageable, but repeated factorization may be expensive. Because the Hessian matrix can be indefinite or singular, an unmodified Newton step need not decrease the loss; a modified Hessian, damping or a trust region is appropriate.

Full BFGS uses gradients and an $O(d^2)$ curvature approximation, avoiding explicit second derivatives. Its rank-two updates and direction calculation cost $O(d^2)$ per step and require roughly the same matrix storage. A positive-definite initial approximation with safeguarded curvature updates and a suitable line search preserves descent directions. For this scale it is often a good compromise, without guaranteeing a global minimum.^[ref]

Differentiating the gradient term by term gives the Hessian matrixTake row $i$ of $A$ to be ${\mathbf{a}}_i^{\mathsf{T}}$. ThenIndeed, entry $(j,k)$ of this product is ${\sum }_i{\Delta }_{ii}{a}_{ij}{a}_{ik}$, exactly the entry of the displayed sum of outer products. Explicitly,These coefficients may have either sign, so a positive-definite Hessian matrix cannot be assumed. This is an instance of the Hessian of a sum of functions of linear forms.

Use a column gradient. By the multivariable chain rule,The scalar coefficients multiply the constant vectors ${\mathbf{a}}_i$. In particular $f^{\prime }(t)=-t{e}^{-t^2/2}$, although explicit differentiation of $f$ is not required.

At $d=10^6$, gradient descent remains a plausible large-scale choice; dense Newton method and full-memory BFGS are generally impractical. A dense $d\times d$ double-precision matrix alone contains $10^{12}$ entries, about $8$ TB. Dense factorization also has order $10^{18}$ arithmetic work. Full BFGS avoids factorization but retains quadratic storage and update work, so it does not solve that scaling problem.

Gradient descent uses linear working storage in $d$, though the cost of reading the data and evaluating the gradient is still $O(nd)$ for dense $A$. Sparsity or streaming can be important. Limited-memory BFGS, a distinct limited-memory variant, stores only a fixed number $m$ of recent update pairs and uses $O(md)$ storage and update work, making it a realistic alternative to full BFGS.^[ref]

The available Hessian matrix factorization also permits a matrix-free Hessian-vector product:It costs $O(nd)$ for dense data without forming $H$. A suitable iterative, safeguarded Newton method can exploit this structure, so the conclusion rules out the dense implementation rather than every possible second-order algorithm.

These methods seek a local solution and require convergence checks appropriate to nonconvex optimization. The formula does not even ensure a finite minimizer exists: with one positive coefficient and nonzero $\mathbf{a}$, the positive exponential loss approaches zero along an unbounded direction without attaining it. Dimension alone cannot supply a convergence guarantee.

Let $h(x,y)=2y^3+x{y}^2-y^2-4x$, so $g=e^h>0$. Its gradient is $\nabla g=g\nabla h$, andThus a stationary point must have $y=\pm 2$ and $x=1-3y$, giving $(-5,2)$ and $(7,-2)$. To classify them, useAt either stationary point, $\nabla h=0$ and $12y+2x-2=6y$. Hence $\det H_g=g^2(-4y^2)=-16g^2<0$. The Hessian has one positive and one negative eigenvalue, so both are saddle points. There are no other stationary points.

For a differentiable scalar function of $n$ real variables, its derivative is the linear mapwhere the column gradient is $\nabla f=(\partial f/\partial x_1,\dots ,\partial f/\partial x_n{)}^T$. Its Hessian is the $n\times n$ matrix $H_f=({\partial }^{2}f/\partial x_i\partial x_j{)}_{i,j}$. For a $C^2$ function, equality of mixed partial derivatives makes this matrix symmetric.

The objective is continuous on the compact disk, so a global maximum exists. Its gradient is $(e^x,2e^{2y})$, which is never zero, so a maximum cannot lie in the interior. It must have $x^2+y^2=1$.

Both coordinates of a maximizing point are positive. Reflecting a negative coordinate to its positive counterpart strictly increases the objective without changing the constraint; at an endpoint with one coordinate zero, a small move along the circle into the positive quadrant also increases the objective to first order. Thus it suffices to consider $x>0,y>0$.

The boundary Lagrange multiplier equations are $e^x=2\lambda x$ and $2e^{2y}=2\lambda y$. Eliminate $\lambda$ to obtain the two-dimensional nonlinear system of equationsKeeping this quadrant restriction excludes the boundary minimum that an unconstrained elimination could also produce. Under the stated uniqueness assumption for the admissible solution, the global maximum must be that solution. Approximate it with multivariable Newton iteration, solving $J_F(z_n){\Delta }_n=-F(z_n)$ and setting $z_{n+1}=z_n+{\Delta }_n$, with a suitable safeguard if needed. For example its explicit Jacobian matrix isNo numerical solution of the system is required.

Use the optimization LagrangianSymmetry gives the gradient equations $2Ax=\lambda 1$ and $2By=\lambda 1$. The positive-definite matrices are invertible, so defineThe affine constraint gives $\lambda =2/D$, henceFor a global certificate, write any other feasible point as $(x_{\ast }+u,y_{\ast }+v)$. Feasibility implies $1^{T}u+1^{T}v=0$. Expansion cancels the cross terms and leavesBy positive definiteness this is positive unless $u=v=0$. Thus the solution is the unique global minimum, not merely a stationary point. This is a two-block instance of positive quadratic minimization under one affine constraint.

Differentiate the column gradient $2e^{r^{T}r}r$. The Jacobian matrix product rule gives one term from differentiating the scalar exponential function and one from differentiating $r$, whose Jacobian matrix is the identity matrix:In particular, for any nonzero vector $v$,Thus the Hessian matrix is positive definite everywhere. This calculation proves strict convexity and supplies the derivatives needed for sums of these terms.

Write $r=x-a$. The squared Euclidean norm is $r^{T}r$, whose gradient is $2r$. The chain rule for the exponential function therefore gives the column gradientThe row derivative, if that convention is used for $df/dx$, is its transpose.

Let $T(x)$ be the weighted average on the right, and define $F$ and $\Phi =\log F$ as in part (i). Their explicit gradient identity isTherefore the displayed equation is exactly the stationary point equation for the strongly convex $\Phi$. It has one solution, which is the unique minimizer from part (i). Reuse the BFGS iteration, withThe gradient is explicit, the weights can again be scaled by their largest exponent, and the usual full-memory BFGS assumptions give local superlinear convergence.

Simply setting $x_{n+1}=T(x_n)$ is not automatically a convergent fixed-point iteration. Differentiating the normalized weights giveswhose operator norm need not be below $1$. For example, in one dimension with centres $-1,1$, $T(x)=-\tanh (2x)$ and $T^{\prime }(0)=-2$: the desired fixed point is locally repelling for that naive iteration. If exact second derivatives are affordable, multivariable Newton iteration on $x-T(x)=0$ is another valid choice, with Jacobian matrix $I-J_T$ as given above and local quadratic convergence.