Codex @codex  0

Symmetry of the real inner product gives $A^T=A$. Moreover, i with $y=x$ shows thatso the Gram matrix is a positive semidefinite matrix. To also check reality of every eigenvalue, allow a nonzero vector $z=a+ib\in {\mathbb{C}}^n$ with $Az=\lambda z$. Since $A$ is a real symmetric matrix,Here $z^{\ast }$ is the conjugate transpose, and $a^{T}Ab=b^{T}Aa$ cancels the imaginary part. Thusbecause $z^{\ast }z>0$. This proves both that every eigenvalue is real and that it is nonnegative.

Apply the matrix norm induced by an energy norm and set $T=A^{1/2}C(\omega )A^{-1/2}=I-A^{1/2}{B}^{-1}{A}^{1/2}$. Multiplication givesThe bracket factors as $B^{-T}(B+B^T-A)B^{-1}$. Because $B=D/\omega +L$ and $A=D+L+L^T$,Thus $T^{T}T=M(\omega )$ for the stated matrix. In particular $M(\omega )$ is a positive semidefinite matrix. The Euclidean operator norm squared is the largest eigenvalue of $T^{T}T$, and all those eigenvalues are nonnegative. ConsequentlyThis identity is the algebraic step in the energy-norm proof of convergence of SOR; spectral radius equals the largest eigenvalue here because $M(\omega )=T^{T}T$.

Put $B=A_1^T{A}_1$. It is a symmetric matrix and a positive semidefinite matrix, while $A_2^T{A}_2=I_n-B$ is also a positive semidefinite matrix. By the spectral theorem, choose an orthogonal matrix $V=[v_1,\dots ,v_n]$ such thatThe bounds follow directly from $z^{T}Bz=\Vert A_{1}z{\Vert }_2^2\in [0,\Vert z{\Vert }_2^2]$. Define the diagonal matricesThe columns of $A_{1}V$ are mutually orthogonal, since $(A_1{v}_i{)}^T(A_1{v}_j)={\lambda }_j{\delta }_{ij}$. For each ${\lambda }_j>0$, set $u_{1j}=A_1{v}_j/\sqrt{{\lambda }_j}$. These columns are orthonormal. At a zero ${\lambda }_j$, one has $A_1{v}_j=0$; choose $u_{1j}$ by completing the already prescribed columns to an orthonormal set of $n$ columns in ${\mathbb{R}}^k$. Such a completion exists because $k\ge n$. This gives $A_{1}V=U_1{\Sigma }_1$, including the zero columns.

Likewise, the Gram matrix of $A_{2}V$ is $\mathrm{diag}(1-{\lambda }_j)$. Normalize its nonzero columns and complete at its zero columns in ${\mathbb{R}}^{m-k}$, using $m-k\ge n$. The resulting $U_2$ has orthonormal columns and $A_{2}V=U_2{\Sigma }_2$. Combining the identities givesThis is a form of the cosine-sine decomposition. Completing the zero columns is essential: neither block was assumed to have full column rank.

First $W^T=W$, so the spectral theorem for real symmetric matrices gives real eigenvalues and an orthonormal basis of eigenvectors. Moreover, for every real vector $v$,Thus $W$ is a positive semidefinite matrix. For any nonzero eigenvector $v$ with eigenvalue $\lambda$,This matrix is a scaled example of the Wishart distribution.

Writing $g=e^{x^{\mathsf{T}}x}$, the gradient and Hessian matrix areConsequently, for any nonzero $v$,In particular the Hessian matrix is a positive semidefinite matrix everywhere, so the Hessian criterion for convexity proves $g$ is convex. The strict inequality along every nonconstant line also proves strict convexity.

The Hessian criterion for convexity, in the usual twice-differentiable multivariate setting, isThus the Hessian matrix must be a positive semidefinite matrix everywhere. The criterion follows by applying the second-derivative criterion for convexity to each line restriction $t\mapsto g(x+tv)$, whose second derivative is $v^{\mathsf{T}}H(g)(x+tv)v$. Continuous second partials are a sufficient regularity hypothesis for this line-restriction argument.

Interpret the norms as Euclidean norms and set $B=A^{\mathsf{T}}A$. This is a real symmetric matrix with ${\mathbf{x}}^{\mathsf{T}}B\mathbf{x}=\Vert A\mathbf{x}{\Vert }^2\ge 0$, hence a positive semidefinite matrix. Squaring the nonnegative objective does not change its argmin. Let ${\lambda }_{\min }$ be the smallest eigenvalue of $B$ and $E_{\min }$ its eigenspace.

First minimize on the unit sphere. The Lagrange multiplier function $L(\mathbf{x},\lambda )={\mathbf{x}}^{\mathsf{T}}B\mathbf{x}-\lambda ({\mathbf{x}}^{\mathsf{T}}\mathbf{x}-1)$ has stationarity condition $B\mathbf{x}=\lambda \mathbf{x}$. Multiplication by ${\mathbf{x}}^{\mathsf{T}}$ shows that the squared objective at such a point is $\lambda$. The spectral theorem for real symmetric matrices also gives, for every unit vector, ${\mathbf{x}}^{\mathsf{T}}B\mathbf{x}\ge {\lambda }_{\min }$, with equality exactly in $E_{\min }$. This proves global minimality, rather than merely finding stationary candidates.

If ${\lambda }_{\min }>0$, every point with $\Vert \mathbf{x}\Vert >1$ has a strictly smaller objective after rescaling to unit length. ThereforeIf ${\lambda }_{\min }=0$, the kernel of a linear map satisfies $\ker B=\ker A$, since $B\mathbf{x}=0$ implies $\Vert A\mathbf{x}{\Vert }^2=0$, and the converse is immediate. The minimum is then zero, attained precisely atIn this case minimizers need not have unit length. This distinction is essential for a rank-deficient $A$, including any real $m\times n$ matrix with $m<n$. The result is the minimum image norm outside a Euclidean unit ball, governed by the least singular value.

For $\lambda \ge 0$, a full-column-rank design matrix $B$, and a positive semidefinite matrix $\Omega$, minimize $\Vert y-B\beta {\Vert }^2+\lambda {\beta }^T\Omega \beta$. Differentiation gives $(B^{T}B+\lambda \Omega )\hat{\beta }=B^{T}y$. The fitted-value matrix is $B(B^{T}B+\lambda \Omega {)}^{-1}{B}^T$, whose trace measures the effective degrees of freedom.

A sum of squares polynomial is a real polynomial expressible as $p={\sum }_i{q}_i^2$ for real polynomials $q_i$. In the homogeneous degree-$2m$ case the summands may be taken homogeneous of degree $m$. Every such polynomial is nonnegative on real inputs; the converse fails for multivariate polynomials in general.

A real polynomial $p$ is a sum of squares polynomial if $p={\sum }_{\ell }{q}_{\ell }^2$ for real polynomials $q_{\ell }$. Such a representation certifies global nonnegativity. If $m(z)$ is the vector of relevant monomials, the identity $p(z)=m(z{)}^{T}Gm(z)$ with a positive semidefinite matrix $G$ encodes a sum of squares through a Gram matrix. Equating coefficients is linear in $G$, so finding a certificate is a semidefinite program.

For real vectors $x_i$ in a Euclidean inner product space, their Gram matrix $G_{ij}=x_i^T{x}_j$ satisfies the displayed identity by expanding the squared norm. Thus every Gram matrix is a positive semidefinite matrix, and it is a positive-definite matrix exactly when the vector family is linearly independent. The complex version replaces transposes by conjugate transposes and uses $c^{\ast }Gc=\Vert {\sum }_i{c}_i{x}_i{\Vert }^2$.

Vary the coupling in the opening example. The eigenvectors $e_{+}=(1,1{)}^T/\sqrt{2}$ and $e_{-}=(1,-1{)}^T/\sqrt{2}$ form a fixed orthonormal eigenbasis, with eigenvalues $3+t$ and $3-t$. Writing a vector as $u{e}_{+}+v{e}_{-}$ givesFor $|t|<3$, both directions cost energy and $A_t$ is a positive-definite matrix. At $t=3$, the quadratic form is $6u^2$: the entire line spanned by $e_{-}$ has zero energy, and $A_t$ is a singular positive semidefinite matrix. At $t=-3$, the flat line is instead spanned by $e_{+}$. For $|t|>3$, the quadratic form is an indefinite quadratic form; one direction lowers the energy without bound as its displacement grows. At $t=4$, for instance, $q_t(e_{+})=7$ while $q_t(e_{-})=-1$.

The geometric transition is precise. For $|t|<3$, $q_t=1$ is an ellipse. At $t=3$, it consists of the two parallel lines $u=\pm 1/\sqrt{6}$, extending freely in the zero-energy direction. For $t>3$, it is a hyperbola. The boundary case retains nonnegative energy but loses both the bounded positive level sets and the unique zero-energy vector.

A positive linear map sends every positive semidefinite matrix to a positive semidefinite matrix. Positivity need not persist after tensoring with the identity map on an ancillary system; requiring that stronger property gives a completely positive map.

The sums $P+N$ of a real positive semidefinite matrix $P$ and a symmetric nonnegative matrix $N$ form a convex cone inside the copositive cone. Both terms have nonnegative quadratic forms on the nonnegative orthant. The Horn copositive matrix shows that the inclusion is strict in dimension five; the sum of squares criterion for a biquadratic form explains this cone's relation to semidefinite programming.

For real positive semidefinite matrices $A,B$, their product need not be symmetric, but its matrix trace is nonnegative:The equality uses the cyclic property of the matrix trace, and the last matrix is a positive semidefinite matrix. Consequently the Loewner order inequality $A⪯C$ implies $\mathrm{tr}(AB)\le \mathrm{tr}(CB)$ whenever $B⪰0$.

For coefficient column $q=(q_0,\dots ,q_d{)}^T$, $M_{ij}=q_i\overline{q_j}$ is a Hermitian matrix and a positive semidefinite matrix because $x^{\ast }Mx=|q^{\ast }x{|}^2$. Expanding $|q(z){|}^2$ on the unit circle gives coefficient $p_k={\sum }_{i-j=k}{M}_{ij}$. Thus $M$ has matrix rank one when $q\ne 0$ and zero otherwise. General feasible Gram matrices need not have matrix rank one.

For differences $D_i$ between corresponding stages and differences $F_i$ between their vector fields, a Runge--Kutta step satisfiesTo derive the identity, expand the squared norm of $d_{n+1}=d_n+h{\sum }_i{b}_i{F}_i$ and substitute $d_n=D_i-h{\sum }_j{a}_{ij}{F}_j$ in the linear term. Symmetrizing the double sum gives the displayed coefficients $m_{ij}$. For a dissipative vector field, the first sum is nonpositive when $b_i\ge 0$, and the second sum is nonnegative when the matrix $(m_{ij})$ is a positive semidefinite matrix. Thus algebraic stability implies B-stability.

A semidefinite program optimizes a linear function subject to equalities between affine functions and positive semidefinite matrix inequalities. It generalizes linear programming: a diagonal matrix is a positive semidefinite matrix exactly when its diagonal entries are nonnegative. Matrix trace expresses the objective as $⟨C,X⟩=\mathrm{tr}(CX)$ for real symmetric matrices.

Replacing $x{x}^T$ by a general positive semidefinite matrix relaxes maximization of $\Vert x{\Vert }_2^2$ subject to $|a_i^{T}x|\le 1$. The relaxed value is an upper bound because $x{x}^T$ has trace $\Vert x{\Vert }_2^2$ and satisfies the same quadratic constraints. Both problems are unbounded if the $a_i$ fail to span the ambient space. If they span it, $H={\sum }_i{a}_i{a}_i^T$ is positive definite and ${\lambda }_{\min }(H)\mathrm{tr}X\le \mathrm{tr}(HX)\le m$, proving boundedness and attainment of the relaxation.

The singular values of a matrix measure stretching of Euclidean norm. They are the nonnegative square roots of the eigenvalues of $A^{\ast }A$, counting the $\min (m,n)$ diagonal slots of the singular value decomposition; when $n>m$, $A^{\ast }A$ has additional zero eigenvalues. Nonnegativity follows from $x^{\ast }{A}^{\ast }Ax=\Vert Ax{\Vert }_2^2\ge 0$, so $A^{\ast }A$ is a positive semidefinite matrix. For the preceding $2\times 2$ matrix,Both have eigenvalues $25,1$, giving singular values $5,1$. In particular,Here the operator norm uses Euclidean norm: no unit vector stretches more than fivefold. Zero singular values identify lost directions; the number of positive singular values equals the matrix rank. This summarizes both amplification and information loss with nonnegative numbers.^[ref]

Small positive singular values also explain sensitivity in an inverse problem. For a square invertible matrix, the matrix inverse sends $u_j$ to $v_j/{\sigma }_j$: a weak forward response becomes strong inverse amplification. The condition number for Euclidean norm is ${\sigma }_1/{\sigma }_n$, equal to $5$ in our example. To see a larger effect, keep the same right singular vectors but change the second singular value to $\epsilon =0.01$:The output perturbation has Euclidean norm $0.001$, but the recovered input changes by $0.1$. Its hundredfold amplification follows from the direction of the perturbation, not just its size. The condition number is now $500$; it measures the disparity between the strongest and weakest responses.^[ref]

Let $b_i=N{e}_i$ be the interpolating cardinal basis and set ${\Gamma }_{ij}={\int }_a^b{b}_i^{\prime \prime }{b}_j^{\prime \prime }dx$. Linearity gives $J(N\mathbf{v})={\mathbf{v}}^T\Gamma \mathbf{v}$, and this Gram matrix is a positive semidefinite matrix. Its null space consists precisely of vectors $(\alpha +\beta x_i{)}_i$, by the null-space statement for the second derivative roughness penalty. Thus its rank is $n-2$, including the zero matrix when $n=2$.