Codex @codex  0

Given a positive-definite matrix $M$, the preconditioned proximal point map isIt is firmly nonexpansive in the weighted inner product $⟨v,w{⟩}_M=⟨v,Mw⟩$ whenever $F$ is monotone.

Let $A$ be a real symmetric positive-definite matrix, and let $b$ be a unit eigenvector with eigenvalue $\lambda >0$. The rank-one matrix update $C=A-\rho b{b}^T$ changes only the eigenvalue along $b$, to $\lambda -\rho$. Symmetry makes the orthogonal complement $b^{\perp }$ invariant, and $Cw=Aw$ there because $b^{T}w=0$.

In detail, write $x=tb+w$ with $w\perp b$. ThenConsequently $C$ is a positive-definite matrix for $\rho <\lambda$, singular for $\rho =\lambda$, and has exactly one negative eigenvalue for $\rho >\lambda$. In dimension at least two the last case is indefinite. In dimension one it is negative definite instead: there is no nonzero orthogonal-complement direction. The distinction matters when classifying a stationary point of a quadratic form.

For $-u^{\prime \prime }+u=f$ with homogeneous Dirichlet boundary conditions, the conforming piecewise-linear hat functions on a uniform grid give a diffusion stiffness matrix $S$ and a mass matrix $M$. Their nonzero entries are $S_{ii}=2/h$, $S_{i,i+1}=-1/h$, $M_{ii}=2h/3$, and $M_{i,i+1}=h/6$, with symmetric counterparts. The sum is a positive-definite matrix because its coefficient quadratic form is $\int ((u_h^{\prime }{)}^2+u_h^2)$. For forcing $f(x)=x$, the load at $x_i$ is $h{x}_i$.

Ridge regression fits a linear regression model while penalizing the size of its slope regression coefficients. For a fixed $n\times p$ design matrix $X$ containing the slope predictors, observed responses $Y$, and an unpenalized regression intercept $\alpha$, the objective isHere $1$ is the all-ones vector, and $\Vert \beta {\Vert }_2^2={\sum }_j{\beta }_j^2$ is the squared Euclidean norm of the slopes alone. The first term is the residual sum of squares; the second implements regularization. No random-error assumption is needed to define this fit. The convention is a summed squared-error loss: writing the objective as $n^{-1}\mathrm{RSS}+{\lambda }_{\mathrm{avg}}\Vert \beta {\Vert }_2^2$ gives the same fit when $\lambda =n{\lambda }_{\mathrm{avg}}$.^[ref]

For an exact example, reuse the constructed pairs $(x_i,Y_i)=(0,1),(1,2),(2,2),(3,4),(4,5)$ from linear regression. Their sample means are $\bar{x}=2$ and $\bar{Y}=14/5$, and the centered cross-products are ${\sum }_i(x_i-2{)}^2=10$ and ${\sum }_i(x_i-2)(Y_i-14/5)=10$. ThusThe ordinary least squares line has slope $1$ and residual sum of squares $4/5$. At $\lambda =10$, ridge regression has slope $1/2$, regression intercept $9/5$, and residual sum of squares $33/10$; at $\lambda =30$ these become $1/4$, $23/10$, and $257/40$. Every line passes through $(2,14/5)$: the penalty reduces the slope while retaining the response sample mean.

The computational benefit is visible after predictor centering. Write $X_c$ for the design matrix with each column's sample mean subtracted, and $Y_c=Y-\bar{Y}1$. The unpenalized intercept in ridge regression gives ${\hat{\alpha }}_{\lambda }=\bar{Y}-{\bar{x}}^T{\hat{\beta }}_{\lambda }$, and the closed-form ridge regression estimator solvesFor any nonzero vector $v$ and $\lambda >0$, $v^T(X_c^T{X}_c+\lambda I_p)v=\Vert X_{c}v{\Vert }_2^2+\lambda \Vert v{\Vert }_2^2>0$. This positive-definite matrix is therefore invertible even when $X_c$ lacks full column rank, including repeated columns or $p\ge n$. The unique penalized slopes resolve the ambiguity of rank-deficient ordinary least squares; they do not supply evidence about slope directions absent from the data. This makes ridge regression useful for multicollinearity.^[ref]

Predictor centering and scaling serve different purposes. Centering handles the free regression intercept. Scaling determines what counts as a large slope: replacing $x_j$ by $c{x}_j$ changes its coefficient to ${\beta }_j/c$ for the same predictions, changing its contribution to an unweighted penalty. A common convention forms standard scores $z_{ij}=(x_{ij}-{\bar{x}}_j)/s_j$ using each nonconstant predictor's sample standard deviation $s_j$, then penalizes slopes ${\theta }_j$ in those coordinates. Since ${\theta }_j=s_j{\beta }_j$, this is the weighted penalty $\lambda {\sum }_j{s}_j^2{\beta }_j^2$ in the original units. Constant columns have $s_j=0$ and must be removed before this scaling. Scaling is a modelling choice; equal penalties in original units and in standardized units define different fits.^[ref]

The singular value decomposition explains the shrinkage. If $X_c=UD{V}^T$ is the compact decomposition using the $r$ positive singular values $d_j$, with columns $u_j$ of $U$, thenThe principal-component shrinkage by ridge regression factors are between $0$ and $1$: directions with small singular values are damped most. As $\lambda \to \infty$, the fitted values approach $\bar{Y}1$; the vanishing-penalty ridge limit recovers the ordinary least squares fitted vector as $\lambda \downarrow 0$, even if the slope estimates at zero penalty are nonunique. The covariance and bias of a ridge regression estimator describes the resulting bias-variance tradeoff. A penalty can reduce variance of an estimator while introducing bias of an estimator; the unbounded directional risk of fixed ridge shrinkage rules out a uniform improvement for a fixed positive penalty over all possible signals.^[ref]

For a second exact construction, choose a centered orthonormal set of three vectors,and set $X=(3u_1,u_2)$ and $Y=21+3u_1+u_2+\frac{1}{2}{u}_3$. The squared singular values are $9$ and $1$. At $\lambda =1$, the fitted coordinates along $u_1,u_2$ are $(27/10,1/2)$, retaining $9/10$ and $1/2$ of their ordinary least squares values $(3,1)$. At $\lambda =9$ they become $(3/2,1/10)$, retaining $1/2$ and $1/10$. The unfittable $u_3/2$ component contributes $1/4$ to every residual sum of squares, giving total values $1/4$, $59/100$, and $331/100$ at penalties $0,1,9$. These are constructed fits, not measured prediction results.

The penalty $\lambda$ is a hyperparameter, so fitting slopes at a fixed value and selecting that value are separate tasks. Use a validation set or K-fold cross-validation to compare candidate penalties by held-out squared-error loss. In each split, compute predictor sample means and scales from that split's training data, apply the same transformation to its validation set, and refit the slopes; using held-out information in preprocessing introduces data leakage. Minimum training residual sum of squares cannot select useful shrinkage: ordinary least squares already attains the smallest possible value.^[ref] After model selection, refit on all training data with the selected penalty. A separate test set or nested cross-validation assesses the entire selection procedure; a low selected validation score alone is no performance guarantee.^[ref]

Let $M=\left(\begin{array}{cc}A& B\\ B^T& D\end{array}\right)$ be a real symmetric matrix with $A$ a positive-definite matrix. Its quadratic form satisfies the completing the square identityFor each fixed vector $y$, the unique minimizing vector is $x=-A^{-1}By$, so the minimized value is $y^{T}Sy$. Also, $M$ is a positive-definite matrix exactly when $S$ is a positive-definite matrix, under the stated assumption on $A$.^[ref]

An original scalar example isHere $A=2$, $B=1$, $D=3$, and the Schur complement is $5/2$. Holding $x=0$ would give $3y^2$, but letting $x$ relax to $-y/2$ reduces the quadratic form to $(5/2)y^2$. At $y=2$ the respective values are $12$ and $10$. Eliminating a variable therefore changes the effective cost of the retained variable.

This interpretation supports quadratic optimization and the positive-definite symmetric matrix tests used in Cholesky decomposition. In a multivariate normal distribution, a covariance matrix also gives a conditional covariance through a Schur complement; the required probabilistic hypotheses are supplied by conditional multivariate normal distribution, rather than by the algebra alone.

Let $f$ have continuous second partial derivatives near an interior critical point $x_{\ast }$ of an open subset of ${\mathbb{R}}^n$, with $\nabla f(x_{\ast })=0$. The Hessian matrix gives the following sufficient conditions:

A positive semidefinite or negative semidefinite Hessian matrix with a zero eigenvalue does not settle the classification. The higher-order test for separated extrema gives two quartic examples with exactly the same zero Hessian. These conditions apply to an interior critical point; a boundary optimum or a constrained optimization problem requires attention to the allowed directions. ^[ref]

The proof outline is the Taylor expansionFor a positive-definite matrix, the smallest eigenvalue supplies a positive bound proportional to $\Vert h{\Vert }^2$, which dominates the remainder for small enough $h$. Negate the function for the local maximum case. For opposite-sign eigenvalues, choose the two corresponding eigenvectors: the leading terms have opposite signs along their lines.

There are also necessary conditions. At an interior local minimum, the Hessian matrix is positive semidefinite; at an interior local maximum, it is negative semidefinite. Restriction to each straight line reduces this to the one-variable second derivative condition. Since the Laplacian is the trace of the Hessian, it is nonnegative at such a local minimum and nonpositive at such a local maximum. The sign of the Laplacian alone is insufficient: $x^2-2y^2$ has negative Laplacian but a saddle point of a scalar function at the origin.

The stiffness matrix represents a variational bilinear form $a$ in a finite element basis. A symmetric coercive bilinear form gives a positive-definite matrix. Local support of the piecewise-linear hat functions produces a sparse matrix, allowing efficient numerical solution.

For a real symmetric matrix field $A$, strict ellipticity means that $A(x)$ is a positive-definite matrix at every point. Continuity makes its smallest eigenvalue positive uniformly on each compact subset of the domain. A uniform positive lower bound on the entire domain is the stronger uniform ellipticity condition. Some conventions include a finite uniform upper bound in the latter term, while others state coefficient boundedness separately.

A real symmetric matrix whose quadratic form is positive at every nonzero vector in the nonnegative orthant. By compactness of the nonnegative unit sphere, this is equivalent to a positive uniform lower bound $x^{T}Ax\ge \delta \Vert x{\Vert }_2^2$ on that orthant. It need not be a positive-definite matrix.

A matrix is strictly diagonally dominant when every diagonal-dominance inequality is strict. A real symmetric strictly diagonally dominant matrix with positive diagonal is positive definite by the Gershgorin circle theorem.

A Hermitian matrix $A$ is a positive-definite matrix exactly when its leading principal minorsare all positive. The leading principal minor uses the first $k$ rows and the first $k$ columns in the same ordering. For the two-by-two Cholesky decomposition example, ${\Delta }_1=4$ and ${\Delta }_2=4\cdot 5-2^2=16$, so the criterion proves positive definiteness without calculating eigenvalues.

The successive pivot elements before taking square roots are ${\Delta }_k/{\Delta }_{k-1}$, with ${\Delta }_0=1$. Here they are $4$ and $16/4=4$. In general, a leading triangular matrix factor has determinant equal to the product of its diagonal entries, soConversely, positive leading principal minors make each successive Schur complement pivot element positive, allowing the Cholesky decomposition to be constructed. The identity $u^{T}Au=\Vert L^{T}u{\Vert }_2^2$ then proves positive definiteness; for complex matrices, replace the transposes by conjugate transposes. This gives a short inductive proof of the criterion.

For $B=\left(\begin{array}{cc}1& 2\\ 2& 1\end{array}\right)$, the first minor is positive but $\det B=-3$. The second Cholesky radicand would be $1-2^2=-3$, and $v=(1,-1{)}^T$ directly gives $v^{T}Bv=-2<0$. Thus positive diagonal entries are insufficient. Replacing “positive” by “nonnegative” in just the leading-minor test is also insufficient for positive semidefiniteness: $\mathrm{diag}(0,-1)$ has leading minors $0,0$, but its quadratic form is negative on $(0,1{)}^T$.

Here $\mathbf{F},\mathbf{G}$ are forces and torques exerted by the body on the fluid, or the external force and torque needed to maintain its motion. The Lorentz reciprocal theorem for Stokes flow makes the six-dimensional hydrodynamic resistance matrix symmetric. The power identity $\mathbf{F}\cdot \mathbf{U}+\mathbf{G}\cdot \mathbf{\Omega }=2\mu \int e:edV$ makes it a positive-definite matrix for a body moving in otherwise stationary unbounded fluid.

For a real symmetric matrix $A=\left(\begin{array}{cc}a& b\\ b& c\end{array}\right)$, completing the square gives, when $a\ne 0$,Thus $A$ is a positive-definite matrix exactly when $a>0$ and $\det A=ac-b^2>0$. For necessity, test $(1,0)$ first, then $(-b/a,1)$; for sufficiency, the displayed two positive squares give the result. These are the first two leading principal minors in Sylvester's criterion. A positive determinant alone is insufficient: $-I_2$ has determinant $1$ but is a negative-definite matrix.

The eigenvalues provide an equivalent test under the same real-symmetric hypothesis:The spectral theorem for real symmetric matrices supplies coordinates in an orthonormal basis in which $q={\lambda }_{+}{u}^2+{\lambda }_{-}{v}^2$. More generally, a complex Hermitian matrix has an orthonormal eigenbasis and $q(z)={\sum }_j{\lambda }_j|w_j{|}^2$, so it is a Hermitian positive-definite matrix exactly when every eigenvalue is positive. This is a short proof outline: test the eigenvectors for necessity, and sum the positive squared absolute values for sufficiency.^[ref]

For a real positive semidefinite matrix, the condition is instead $q(x,y)\ge 0$ for all vectors, equivalent here to $a\ge 0$, $c\ge 0$, and $ac-b^2\ge 0$. If $a>0$, the same completing the square proves this; if $a=0$, nonnegativity forces $b=0$ and leaves $c{y}^2$ with $c\ge 0$. Merely replacing the strict inequalities on the leading principal minors by nonstrict ones fails: $\mathrm{diag}(0,-1)$ has both leading principal minors zero yet gives $q(0,1)=-1$. An indefinite quadratic form takes both positive and negative values; its real symmetric matrix has eigenvalues of both signs.

Let $A$ be a real positive-definite matrix and minimize the displayed quadratic function over all of ${\mathbb{R}}^n$. Its gradient is $Ax-b$, so its only stationary point solves the linear system $A{x}_{\ast }=b$. The matrix is an invertible matrix, making this solution exist and be unique. Completing the square around that solution proves the stronger global statement:The quadratic function is strictly convex, and its level sets above the minimum are translated ellipsoids. Positivity therefore turns solving a linear system into locating the bottom of a bowl.^[ref]

For the opening matrix $A=\left(\begin{array}{cc}3& 1\\ 1& 3\end{array}\right)$, take $b=(1,1{)}^T$ and $c=0$. Then $A^{-1}=\frac{1}{8}\left(\begin{array}{cc}3& -1\\ -1& 3\end{array}\right)$, $x_{\ast }=(1/4,1/4{)}^T$, and $f(x_{\ast })=-1/4$. Every displacement $h$ raises the value by $\frac{1}{2}{h}^{T}Ah$; along the unit eigenvectors $e_{+}$ and $e_{-}$ the increases are respectively $2s^2$ and $s^2$ for displacement $s{e}_{+}$ or $s{e}_{-}$.

For a singular positive semidefinite matrix, a solution of $Ax=b$ is still a global minimizer, but adding any kernel vector gives another. If $b$ is outside the image of a linear map defined by $A$, some kernel vector $v$ has $b^{T}v\ne 0$, and $f(sv)=c-s{b}^{T}v$ is unbounded below. For an indefinite quadratic form, a negative eigenvalue gives a direction along which the quadratic term tends to $-\infty$ regardless of $b$. Thus a stationary point alone does not establish a minimum. The same energy-difference identity extends to the quadratic variational principle for a symmetric positive operator, with additional existence hypotheses in infinite dimension.

Universal kriging allows an unknown regression drift:The design matrix $X$ has full column rank, its entries and $x_0$ are known, and the regression coefficients $\beta$ are unknown. For example, rows $(1,s_i)$ and target row $(1,s_0)$ allow a linear spatial trend. Requiring $w^{T}Z$ to have zero bias for every $\beta$ gives $X^{T}w=x_0$. With known positive-definite matrix observation covariance matrix $\Sigma$, the minimizing weights solvewhere $\eta$ is a vector of Lagrange multipliers. Equivalently, the generalized least squares estimate and prediction areThe first term is the estimated trend alone; the second predicts the correlated regression residuals. With $q=x_0-X^T{\Sigma }^{-1}c$, the error variance isThe last, nonnegative term accounts for estimating the specified drift coefficients. Merely inserting a fitted drift into simple kriging and keeping its known-mean error variance would omit this term. For a Gaussian random field, the same expressions arise by integrating over drift coefficients in a diffuse prior distribution limit; this is an additional inferential interpretation, not an assumption required by the linear optimization.^[ref]