Codex @codex  0

Suppose a linear map between finite-dimensional Hilbert spaces has matrix $A$, and their inner products in the chosen bases are $⟨x,z{⟩}_H=z^{\ast }{G}_{H}x$ and $⟨u,v{⟩}_K=v^{\ast }{G}_{K}u$. The Gram matrices $G_H,G_K$ are positive-definite matrices. Writing $A^{\ast }={\overline{A}}^{\mathrm{T}}$ for the ordinary conjugate transpose, the coordinate matrix of the adjoint operator isIndeed, the defining inner product identity says $y^{\ast }{G}_{K}Ax=y^{\ast }[T^{\ast }{]}^{\ast }{G}_{H}x$ for all $x,y$. Thus simply transposing and conjugating is valid in orthonormal bases, where both Gram matrices are the identity matrix.

For the weighted inner product $⟨x,y{⟩}_G=\overline{y_1}{x}_1+4\overline{y_2}{x}_2$ and the coordinate swap $A=\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)$, the adjoint operator has matrix $\left(\begin{array}{cc}0& 4\\ 1/4& 0\end{array}\right)$. With $x=(1,0{)}^{\mathrm{T}}$ and $y=(0,1{)}^{\mathrm{T}}$, both inner products equal $4$. The coordinate swap is its own ordinary transpose, but is not self-adjoint in this weighted inner product.

A best linear unbiased estimator minimizes covariance among linear estimators unbiased for the same target. In $Y=X\beta +\epsilon$, assume $\mathbb{E}\epsilon =0$, $X$ is a full-column-rank design matrix and the known covariance matrix $\Sigma$ is a positive-definite matrix. The best linear unbiased estimator of the coefficient vector is $(X^T{\Sigma }^{-1}X{)}^{-1}{X}^T{\Sigma }^{-1}Y$. A spatial trend estimate excludes the correlated residual prediction included in universal kriging.

Every real symmetric matrix that is a positive-definite matrix has a unique Cholesky decomposition $A=L{L}^T$, where $L$ is a lower triangular matrix with positive diagonal entries.^[ref] For a complex Hermitian positive-definite matrix, the corresponding factorization is $A=L{L}^{\ast }$, with $L^{\ast }$ the conjugate transpose and the diagonal of $L$ positive and real.^[ref] The factorization turns a linear system into two triangular systems.

For example, matching entries in $A=L{L}^T$ givesChoose the positive square roots in order: ${\ell }_{11}=2$, ${\ell }_{21}=1$ and ${\ell }_{22}=2$. ThereforeThe second radicand, $5-1^2=4$, is the scalar Schur complement remaining after eliminating the first variable. A positive diagonal of $A$ alone would not guarantee that this quantity is positive; Sylvester's criterion explains the relevant leading principal minors.

For that right-hand side, first use forward substitution to solve $Ly=b$: $2y_1=8$ and $y_1+2y_2=12$, so $y=(4,4{)}^T$. Then use backward substitution in a triangular system to solve $L^{T}x=y$: $2x_2=4$ and $2x_1+x_2=4$, giving $x=(1,2{)}^T$. No matrix inverse is required. For several right-hand sides with the same $A$, keep the same factor $L$ and repeat only these triangular solves.

André-Louis Cholesky developed his calculation in the setting of geodetic adjustment: measured angles and lengths must satisfy geometric relations, while corrections should be small. The manuscript reproduced in Claude Brezinski's study is titled Sur la résolution numérique des systèmes d’équations linéaires and ends with the date 2 December 1910 at Vincennes. The manuscript reports a 56-equation calculation adjusting altitudes in an Algerian triangulation network. Its date establishes when this written account existed, rather than an exact invention date.^[ref]

A related algebraic reason for the method's usefulness is that normal equations involve a Gram matrix. If a real matrix $M$ has linearly independent columns, thenso $M^{T}M$ is a positive-definite matrix. That gives the structure needed by a Cholesky decomposition. The historical adjustment problem instead produces $M{M}^T$ from independent constraint rows; the same argument applies to its transpose. In numerical least squares, explicitly forming normal equations squares the condition number of a full-column-rank design matrix; a QR decomposition can therefore be preferable for accuracy. Indeed, the extreme singular values give ${\kappa }_2(M^{T}M)={\sigma }_{\max }(M{)}^2/{\sigma }_{\min }(M{)}^2={\kappa }_2(M{)}^2$. The factorization remains useful when a positive-definite matrix is already the natural input.

For $\lambda >0$, the ridge regression objective $\Vert Y-X\beta {\Vert }_2^2+\lambda \Vert \beta {\Vert }_2^2$ has gradient $2(X^{T}X+\lambda I)\beta -2X^{T}Y$. The matrix $X^{T}X+\lambda I$ is positive definite, so the unique minimizer is ${\hat{\beta }}_{\lambda }=(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$.

A coarse-grid correction improves a fine-grid approximation by solving a smaller equation for the part of its approximation error representable on a coarse mesh. Use the signed convention $e_h=u_h^{\ast }-v_h$ and the linear-system residual $r_h=b_h-A_h{v}_h=A_h{e}_h$. A prolongation linear map $P:{\mathbb{R}}^m\to {\mathbb{R}}^n$ interpolates coarse corrections into fine-grid corrections, and a restriction linear map $R:{\mathbb{R}}^n\to {\mathbb{R}}^m$ transfers the linear-system residual. Solve $A_H{z}_H=R{r}_h$ exactly or approximately and update $v_h\leftarrow v_h+P{z}_H$. The plus sign follows from defining the approximation error as exact minus approximate. It is the correction equation's right-hand side $R{r}_h$, rather than merely restricted solution values, that determines the update.

For the multigrid method model $A_h=h^{-2}\mathrm{tridiag}(-1,2,-1)$, take $n=2m+1$, so $H=2h$ and the coarse nodes coincide with every second fine node. Use linear interpolation with zero boundary corrections $z_0=z_{m+1}=0$:Use full-weighting restriction for the interior linear-system residual:Here $P^T$ is the matrix transpose in ordinary unweighted coordinates. With these specific transfers, the Galerkin coarse-grid operator isThus it equals the rediscretized Poisson equation operator on this constant-coefficient one-dimensional mesh. The equality is special to the stated model and transfers. In particular, dropping $h^{-2}$ from the fine matrix while using $H^{-2}$ on the coarse mesh changes the correction unless the right-hand sides and transfers are rescaled consistently.

Take seven interior points, $h=1/8$, three coarse unknowns, and the deliberately constructed approximation errorFor example, choose $b_h=0$, so $u_h^{\ast }=0$ and $v_h=-e_h$. Direct matrix multiplication givesThe exact coarse solve returns $z_H=z$, so adding $P{z}_H$ removes the entire chosen approximation error. A broad triangular error can have a sharply localized linear-system residual: the latter measures discrete curvature, not error height. This is why the two should not be confused. A smooth sine mode instead has linear-system residual ${\lambda }_k(A_h)e_h$, so interpreting its magnitude also requires the eigenvalue and the equation's scaling.

The algebra explains the exact cancellation beyond this example. Assume $P$ has linearly independent columns and choose an invertible Galerkin coarse-grid operator $A_H=R{A}_{h}P$. With an exact coarse solve, the iteration matrix for the remaining approximation error isThe short proof is to substitute $R{A}_{h}P=A_H$. Thus every vector in the column space of $P$ is eliminated, and the corrected linear-system residual restricts to zero. These are exact projection identities, not a statement that every visually smooth vector belongs to that column space. If $A_h$ is a symmetric matrix and a positive-definite matrix, and $R=c{P}^T$ with $c>0$, then $A_H$ is also a positive-definite matrix. Moreover $P^T{A}_{h}C{e}_h=0$: $C$ is an orthogonal projection onto the complement of the column space of $P$ in the energy norm $\Vert q{\Vert }_{A_h}=\sqrt{q^T{A}_{h}q}$. Equivalently, the added correction is the best approximation to $e_h$ from that column space in this energy norm. Without these symmetry and transfer assumptions, the projection need not be orthogonal.^[ref]

For an approximate coarse solve represented by a fixed linear map $B_H$, replace $A_H^{-1}$ by $B_H$. ThenThere is no exact cancellation of coarse-representable approximation error unless $B_H{A}_H=I$ on the relevant directions. For the triangular example, deliberately returning half the exact correction, $B_H=\frac{1}{2}{A}_H^{-1}$, leaves $e_h/2$, as the figure shows. This controlled mathematical comparison illustrates an incomplete coarse solve; it is not a measured performance result or a prescription for a geometric multigrid V-cycle. More generally, if $z_H^{\ast }=A_H^{-1}R{r}_h$, then an approximate $z_H$ leaves $C{e}_h+P(z_H^{\ast }-z_H)$, separating the nonrepresentable remainder from the coarse-solve defect.

With pre- and post-relaxation iteration matrices $S_{\mathrm{pre}}$ and $S_{\mathrm{post}}$, a two-grid error step has iteration matrix $E=S_{\mathrm{post}}{C}_B{S}_{\mathrm{pre}}$. A geometric multigrid V-cycle supplies $B_H$ by one recursive coarse cycle from zero, with a direct solve only at the bottom. Effective smoothing, a coarse column space that approximates the surviving approximation error, and a sufficiently accurate coarse correction must work together. Coarsening alone supplies no convergence guarantee, and correcting an arbitrary fine-grid vector exactly would require a coarse column space large enough to contain it.

An ellipsoidal confidence region defined by a positive-definite matrix $A$ and a calibrated threshold $c$. For a known-covariance multivariate normal distribution sample, $A=n{\Sigma }^{-1}$ and the $1-\alpha$ quantile of the chi-squared distribution with $d$ degrees of freedom give exact coverage for the $d$-dimensional mean.

For a positive-definite matrix $A$, the conjugate gradient method chooses mutually $A$-conjugate search directions. Starting with $r_0=b-A{x}_0$ and $p_0=r_0$, it usesand $p_{k+1}=r_{k+1}+{\beta }_k{p}_k$ with
${\beta }_k=r_{k+1}^T{r}_{k+1}/(r_k^T{r}_k)$.

For $X\sim N(\mu 1,C)$ with a known positive-definite matrix $C$, the maximum-likelihood estimator of $\mu$ is $\hat{\mu }=(1^T{C}^{-1}X)/(1^T{C}^{-1}1)$. It is unbiased with variance $(1^T{C}^{-1}1{)}^{-1}$, attaining the Cramér-Rao bound. It is also the best linear unbiased estimator. Pairwise admissible correlation coefficients alone do not ensure that $C$ is a valid covariance matrix; the entire matrix must be symmetric and a positive semidefinite matrix. Inverse-based formulas require positive definiteness.

Take two electrical circuits containing coupled inductors. Let their constant self-inductances be $L_1,L_2>0$, mutual inductance $M$, and positive electrical resistances $R_1,R_2$. Choose current directions so that the off-diagonal entries of the inductance matrix are $+M$. Assume a linear, quasistatic model, with no capacitive effects or core hysteresis. The magnetic flux linkages are ${\lambda }_1=L_1{i}_1+M{i}_2$ and ${\lambda }_2=M{i}_1+L_2{i}_2$. Combining their time derivatives with resistive electric potential differences givesThe secondary is a closed, undriven electrical circuit. Require $M^2<L_1{L}_2$, making $\mathbf{L}$ a positive-definite matrix. The chosen current convention fixes the signs; reversing the secondary's reference direction changes the sign assigned to $M$ and $i_2$ without changing the physical circuit.

There is an essential switching assumption: the primary loop remains closed when $V$ becomes zero. An ideal voltage source then imposes zero electric potential difference, allowing the current to decay. Literally opening an inductive circuit requires additional physics to determine the interruption, including possible capacitance and an electric arc. This closed-return model explains transient induction while keeping that distinction from Faraday's historical making and breaking of contact explicit.

For a real random vector $X=(X_1,\dots ,X_p{)}^T$ with finite second moments, its covariance matrix collects all pairwise covariances:The diagonal entries are variances. The covariance matrix is a symmetric matrix and a positive semidefinite matrix, because for every real vector $a$,Thus one matrix describes the variance of every linear combination of the coordinates. It is a positive-definite matrix exactly when no nonzero linear combination is constant almost surely. A deterministic linear relation produces a zero eigenvalue.

For example,has unit coordinate variances, covariance $0.8$, and eigenvalues $1.8$ and $0.2$. Its unit eigenvectors $(1,1{)}^T/\sqrt{2}$ and $(1,-1{)}^T/\sqrt{2}$ describe directions of greatest and least projected variance. For a unit vector $a=(\cos \theta ,\sin \theta {)}^T$, direct multiplication gives $a^T\Sigma a=1+0.8\sin (2\theta )$: the variation in the sum direction is nine times that in the difference direction.

The spectral theorem supplies an orthonormal basis of these directions. This connects covariance matrices to principal component analysis: its leading eigenvector identifies the unit-direction projection with the largest variance. For a linear transformation $A$, the transformed random vector has covariance matrix $A\Sigma A^T$, so uncertainties propagate through a linear model by matrix multiplication.

For $\lambda >0$ and at least two distinct design points, the unique $C^2[a,b]$ minimizer of squared residuals plus $\lambda J(g)$ is $N(I+\lambda \Gamma {)}^{-1}\mathbf{Y}$. The minimum roughness property of the natural cubic spline interpolant reduces the optimization to the value vector. The matrix $I+\lambda \Gamma$ is a positive-definite matrix, so the normal equations have a unique solution; the interpolation equality case gives uniqueness as a function. Linear tails are part of the natural boundary convention.

Suppose $f$ is a differentiable function on an open convex set $D\subseteq {\mathbb{R}}^n$ and has an $L$-Lipschitz gradient there, in the Euclidean norm. For $u,v\in D$, the descent lemma statesA bounded continuous Hessian matrix gives one sufficient route to this estimate.^[ref]

For a short proof, put $d=v-u$. The fundamental theorem of calculus along the line segment givesThe Cauchy-Schwarz inequality and the Lipschitz gradient bound make the integrand at most $Lt\Vert d{\Vert }_2^2$; its integral is $L\Vert d{\Vert }_2^2/2$. This proof needs no assumption that $f$ is a convex function.

For the gradient descent update $v=u-\eta \nabla f(u)$, provided $v\in D$, substitution yieldsFor $L>0$ and $0<\eta <2/L$, this ensures strict decrease whenever the gradient is nonzero. At $\eta =2/L$ it guarantees only nonincrease; the quadratic example has a nonshrinking alternating component at this endpoint. At $\eta =1/L$ the guaranteed decrease is $\Vert \nabla f(u){\Vert }_2^2/(2L)$. A bound valid only on $D$ cannot justify a step leaving $D$.

If a fixed admissible step size is used forever, every iterate stays in $D$, and $f(x_k)$ has a finite lower bound, summing this inequality shows ${\sum }_k\Vert \nabla f(x_k){\Vert }_2^2<\infty$, hence $\nabla f(x_k)\to 0$. That conclusion alone gives neither convergence of the iterates nor a global minimum. For example, the smooth polynomial $f(x)=x^4-x^2$ has $f^{\prime }(0)=0$ and $f^{\prime \prime }(0)=-2$: gradient descent started at zero never moves, although $f(1/\sqrt{2})=-1/4<0$. On $D=(-1,1)$, $|f^{\prime \prime }(x)|\le 10$, so the Lipschitz gradient hypothesis holds on this domain. Convex functions and strongly convex functions provide additional structure; the exact quadratic convergence result above uses a positive-definite matrix.

Let $A$ be a real positive-definite matrix and let the columns of $U$ be an orthonormal basis of a two-dimensional linear subspace in three-dimensional Euclidean space. Points in that central plane are $x=Uq$, with $|x{|}^2=|q{|}^2$. Restricting the ellipsoid $x^{T}Ax=1$ gives $q^T(U^{T}AU)q=1$. The compressed matrix $U^{T}AU$ is a positive-definite matrix, and its extreme eigenvalues give the distance formulas in the title by the Rayleigh quotient. The two extremal axes are the images under $U$ of the corresponding eigenvectors. This restriction reduces a problem with two Lagrange multipliers to a two-dimensional quadratic form.

For a real symmetric matrix $A$, every stationary squared distance $x^{T}x$ on $x^{T}Ax=1$ occurs in a positive-eigenvalue eigenspace. The Lagrange multiplier equation is $x=\Lambda Ax$; multiplying by $x^T$ gives $\Lambda =|x{|}^2>0$, hence $Ax=|x{|}^{-2}x$. If $A$ is a positive-definite matrix, the nearest and farthest squared distances are $1/{\lambda }_{\max }$ and $1/{\lambda }_{\min }$. With a repeated eigenvalue, the stationary set may contain an entire sphere within its eigenspace. For an indefinite or singular form the surface may be unbounded, and a stationary point need not be a global extremum.

An ellipsoid is a level set $(x-c{)}^{T}A(x-c)=1$ for a real symmetric positive-definite matrix $A$. Its principal axes are the orthonormal eigenvectors of $A$, and an eigenvalue $\lambda$ gives semi-axis length $1/\sqrt{\lambda }$.

For a real positive-definite matrix $A$ in finite dimension, write $A=Q\Lambda Q^T$ by the spectral theorem for real symmetric matrices. In the coordinates in an orthonormal basis $w=Q^{T}x$, the positive level set $q(x)=k$ becomesThis is an ellipsoid, with semi-major and semi-minor axes in two dimensions forming an ellipse; each semi-axis has length $r_j=\sqrt{k/{\lambda }_j}$ and direction given by the corresponding eigenvector. In the opening example, the shorter semi-axis is $\sqrt{k}/2$ along $e_{+}$, while the longer is $\sqrt{k/2}$ along $e_{-}$. A stiffer direction, meaning a larger eigenvalue, reaches the same energy at a smaller displacement. The ratio of longest to shortest semi-axis is $\sqrt{{\lambda }_{\max }/{\lambda }_{\min }}$, the square root of the spectral condition number of a positive-definite matrix.^[ref]

Strict positivity matters because ${\lambda }_{\min }>0$ givesHence a fixed positive level set cannot extend arbitrarily far. The Cholesky decomposition $A=L{L}^T$ gives another view: $q(x)=\Vert L^{T}x{\Vert }^2$, so the ellipsoid is the inverse image of a sphere under an invertible linear map. A singular positive semidefinite matrix has a nonzero kernel; adding a kernel vector preserves its quadratic form, so any nonempty positive level set is unbounded. These statements concern finite dimension; the positive definite symmetric operator entry explains why strict positivity alone need not give a uniform lower bound on an infinite-dimensional Hilbert space.

Let $V_t=V_{t-1}+x_t{x}_t^T$ with positive-definite matrix $V_0$, and suppose $u_t=x_t^T{V}_{t-1}^{-1}{x}_t\le 1$. The matrix determinant lemma gives $\det V_t/\det V_{t-1}=1+u_t$. Since $2\log (1+u)-u$ has nonnegative derivative $(1-u)/(1+u)$ for $0\le u\le 1$, summation proves the displayed bound. In particular, $V_0=\lambda I$, $\lambda \ge 1$, and $\Vert x_t{\Vert }_2\le 1$ ensure $u_t\le 1$. Without that restriction a general form is ${\sum }_t\min \{1,u_t\}\le 2\log (\det V_n/\det V_0)$: for $u\ge 1$, $2\log (1+u)\ge 2\log 2\ge 1$. The bound connects widths of confidence ellipsoids to growth of a determinant.

The magnetic energy of this constant-inductance model isThe positive-definite matrix condition makes $W\ge 0$. Multiplying the two circuit equations by $i_1$ and $i_2$, then adding, gives the displayed conservation of energy statement: input power $V{i}_1$ supplies stored magnetic energy and Joule heating. With zero drive, $\dot{W}\le 0$. After the drive becomes zero, the stored magnetic energy supplies the remaining Joule heating.

For the Jacobi iteration for a three-variable tridiagonal system,An orthonormal eigenbasis iswith eigenvalues $q=\sqrt{2}/4$, $0$, and $-q$, respectively. These are error modes: their shapes persist under the iteration matrix, while their coefficients change. Direct multiplication verifies them, or expansion of the determinant gives $\det (tI-B)=t(t^2-1/8)$. Thus $\rho (B)=q<1$ and convergence holds for every starting vector.

With $x^{(0)}=0$, the initial forward error has coefficientsFor $k\ge 1$,The zero mode disappears after one sweep; the negative mode alternates in sign. The Euclidean norm drops by exactly $q$ per sweep from $k=1$ onward. The first norm drop is different because $\Vert e^{(0)}{\Vert }_2=\sqrt{14}$ includes the zero mode. A starting error proportional to $u_0$ would disappear entirely in one sweep, faster than the spectral radius predicts for generic errors.

The curves are computed iterates and exact formulas for this linear system. Since $A=4(I-B)$, the same eigenvectors have eigenvalues $4-\sqrt{2}$, $4$, and $4+\sqrt{2}$ for $A$. They prove that $A$ is a positive-definite matrix, and give the condition number used in residual and forward error in a stationary iteration.