Codex @codex  0

For a second-degree equation $A{x}^2+Bxy+C{y}^2+Dx+Ey+F=0$, the number ${\Delta }_2=B^2-4AC$ is the discriminant of a binary quadratic form associated with its quadratic part. For a nonempty, nondegenerate real conic section, a negative value gives an ellipse, zero a parabola, and a positive value a hyperbola. It is invariant under a rotation of orthonormal coordinates.^[ref]

A short explanation uses the symmetric matrixAn orthogonal matrix $R$ changes this matrix to $R^{\mathsf{T}}HR$, whose determinant equals $\det H$. The spectral theorem for real symmetric matrices then removes the mixed term. Equal-sign, opposite-sign and one-zero eigenvalues give the three types after the remaining linear terms and offsets are considered.

The qualification matters. The equations $x^2+y^2=0$, $y^2=0$ and $xy=0$ describe a point, a repeated straight line, and two crossing straight lines, respectively. Their quadratic discriminants have the signs expected of ellipse, parabola and hyperbola types, but all three are degenerate conics. Also, $x^2+y^2=-1$ has an empty set of real solutions despite its negative quadratic discriminant.

The continuous-time diffusion model on a finite undirected graph uses the Graph Laplacian in the ordinary differential equationAcross each edge, the higher value sends an amount proportional to the difference to the lower value. The two contributions cancel in the sum over vertices, giving conservation of mass:If the entries represent masses, start with nonnegative entries; signed entries can instead describe deviations from a common level. For the three-vertex path graph, the initial vector $x(0)=(1,0,-1{)}^T$ is an eigenvector with eigenvalue one, so the exact solution isMore generally, an eigenvector with eigenvalue $\lambda$ decays by $e^{-\lambda t}$. The spectral theorem for real symmetric matrices supplies an orthonormal basis of these eigenvectors, and the kernel of a graph Laplacian identifies the surviving terms. On a nonempty connected graph,The limiting value is the original arithmetic mean. On a disconnected graph, each connected component of a graph converges to its own original arithmetic mean. The same matrix therefore measures neighbour disagreement, detects disconnected regions and governs relaxation toward agreement.

For a real positive-definite matrix $A$ in finite dimension, write $A=Q\Lambda Q^T$ by the spectral theorem for real symmetric matrices. In the coordinates in an orthonormal basis $w=Q^{T}x$, the positive level set $q(x)=k$ becomesThis is an ellipsoid, with semi-major and semi-minor axes in two dimensions forming an ellipse; each semi-axis has length $r_j=\sqrt{k/{\lambda }_j}$ and direction given by the corresponding eigenvector. In the opening example, the shorter semi-axis is $\sqrt{k}/2$ along $e_{+}$, while the longer is $\sqrt{k/2}$ along $e_{-}$. A stiffer direction, meaning a larger eigenvalue, reaches the same energy at a smaller displacement. The ratio of longest to shortest semi-axis is $\sqrt{{\lambda }_{\max }/{\lambda }_{\min }}$, the square root of the spectral condition number of a positive-definite matrix.^[ref]

Strict positivity matters because ${\lambda }_{\min }>0$ givesHence a fixed positive level set cannot extend arbitrarily far. The Cholesky decomposition $A=L{L}^T$ gives another view: $q(x)=\Vert L^{T}x{\Vert }^2$, so the ellipsoid is the inverse image of a sphere under an invertible linear map. A singular positive semidefinite matrix has a nonzero kernel; adding a kernel vector preserves its quadratic form, so any nonempty positive level set is unbounded. These statements concern finite dimension; the positive definite symmetric operator entry explains why strict positivity alone need not give a uniform lower bound on an infinite-dimensional Hilbert space.

For an integer $0\le k\le n$, the fantope is the convex hull of rank-$k$ orthogonal projection matrices in ${\mathbb{R}}^n$. Equivalently, it consists of real symmetric matrices with eigenvalues in $[0,1]$ and matrix trace $k$. To prove the equivalence, apply the spectral theorem for real symmetric matrices and express the eigenvalue vector as a convex combination of the zero-one extreme points of the capped simplex. The sum of the largest eigenvalues is its support function.

Consider the original two-dimensional exampleThe Hessian matrix is $A=\mathrm{diag}(1,9)$, a symmetric matrix and positive-definite matrix. Its eigenvectors are the coordinate directions, its eigenvalues are $1$ and $9$, and its Euclidean norm condition number is $9$. The unique global minimum is $x_{\ast }=0$, so the error $e_k=x_k-x_{\ast }$ equals $x_k$. The linear map $I-\eta A$, with $I$ the identity matrix, givesFor $\eta =1/5=0.2$, the two coordinate error multipliers are $0.8$ and $-0.8$. Consequently $x_1=(1.6,-0.8{)}^T$, $x_2=(1.28,0.64{)}^T$, and the second coordinate changes sign at every step while both shrink. The Euclidean norm of the error contracts by $0.8$ each step. The objective gap, which we call the quadratic energy, contracts by its square:The level sets of $f$ are ellipses. Oscillation across their narrow direction is compatible with strictly decreasing energy.

For $\eta =1/4=0.25$, the coordinate error multipliers become $0.75$ and $-1.25$, giving $x_1=(1.5,-1.25{)}^T$. The first coordinate still tends to zero, but the second alternates and grows. Direct substitution givesThus a larger positive step size can turn a strictly convex function into a divergent iteration. Energy and error are different quantities: coordinate errors carry signed multipliers, while each coordinate's contribution to energy carries the squared multiplier.

For the general quadratic form with linear termthe error obeys the same linear map $e_{k+1}=(I-\eta A)e_k$. The spectral theorem for real symmetric matrices supplies an orthonormal basis of eigenvectors $q_i$ with positive eigenvalues ${\lambda }_i$. Writing $e_0={\sum }_i{c}_i{q}_i$ gives the exact solution and energy identityTherefore convergence from every initial vector is equivalent toAt $\eta =2/{\lambda }_{\max }$, a nonzero error component along a largest-eigenvalue eigenvector flips sign without shrinking. Above that threshold it grows. A particular start can still converge if it has zero components in every unstable direction; the interval is the condition for all starts. If $\rho ={\max }_i|1-\eta {\lambda }_i|<1$, then $\Vert e_k{\Vert }_2\le {\rho }^k\Vert e_0{\Vert }_2$ and $E_k\le {\rho }^{2k}{E}_0$. These bounds follow by summing the squared components in the orthonormal basis.

For a positive-definite symmetric matrix $A$, the ellipsoid $x^{T}Ax=1$ has nearest points to the origin exactly at $x={\lambda }_{\max }^{-1/2}u$, where $u$ ranges over unit vectors in the largest-eigenvalue eigenspace. Indeed, the spectral theorem for real symmetric matrices gives $x^{T}Ax\le {\lambda }_{\max }|x{|}^2$, with equality precisely in that eigenspace. A simple largest eigenvalue gives two opposite nearest points; multiplicity $m>1$ gives a scaled $(m-1)$-sphere of nearest points.

The command centres the log measurements and performs principal component analysis of their covariance matrix. With cor=F, it does not first standardize the variables to variance one. For a unit principal component loading vector $q$, the score $q^T(z_i-\bar{z})$ has sample variance $q^T\hat{\Sigma }q$. By the spectral theorem for real symmetric matrices, expand $q$ in an orthonormal eigenbasis; that variance is a weighted average of the eigenvalues. Its maximum is the largest eigenvalue, reached at the corresponding eigenvector. Repeating subject to orthogonality to previous directions gives the other principal components.

The software uses the divisor-$31$ sample covariance matrix for its component standard deviations, whereas var used divisor $30$. Diagonalizing the displayed matrix gives approximatelyTaking square roots of the latter gives $(0.5671603,0.07312405,0.03064834)$, agreeing with the output to the rounding of the displayed sample covariance matrix. The normalization changes these numerical standard deviations but leaves the eigenvectors and proportions of explained variance of a principal component unchanged.

The first principal component explains about $98.08\%$ of the total log-variable variance, the second another $1.63\%$, and the last about $0.286\%$. Keeping two preserves about $99.714\%$. An original scree plot displays the sharp drop after the first component:

Consequently the centered cloud is very elongated along one direction: a one-dimensional approximation represents almost all of its squared variation. From the displayed covariance matrix, a unit first eigenvector, with its arbitrary sign chosen positive, is approximatelyIts principal component score is therefore $0.3980(\log G_i-\overline{\log G})+0.09514(\log H_i-\overline{\log H})+0.9124(\log V_i-\overline{\log V})$. All coefficients have the same sign, so this direction describes overall tree size, with log volume dominating the unstandardized variation. The summary alone supplies proportions, not the loading directions; the displayed sample covariance matrix supplies the latter. This is a geometric approximation, not evidence of an exact deterministic relation or independent original variables.

One further use is principal component analysis on a correlation matrix, obtained with cor=T, which equalizes marginal variances and answers a different scaling-sensitive question. Another is principal component regression: fit a response to selected principal component scores rather than highly correlated original predictors, choosing the number using prediction assessment. The fitted transformation also projects new measurements into the same low-dimensional coordinates, provided the training centering and scaling are retained.

For a real matrix $A$ with positive determinant, the polar decomposition of an invertible real matrix gives unique factorizationswhere $U$ and $V$ are the right stretch tensor and left stretch tensor, respectively. Positivity here means positive definiteness, not merely nonnegative entries.

To prove existence, $C=A^{T}A$ is symmetric and $v^{T}Cv=|Av{|}^2>0$ for every nonzero $v$, because $A$ is invertible. The spectral theorem for real symmetric matrices supplies an orthogonal matrix $Q$ and positive eigenvalues $c_j$ such that $C=Q\mathrm{diag}(c_1,c_2,c_3)Q^T$. Define $U$ using the principal square root of a positive semidefinite matrix construction:Then $R^{T}R=U^{-1}{A}^{T}A{U}^{-1}=I$, so $R$ is orthogonal. Also $\det R=\det A/\det U>0$; an orthogonal matrix has determinant plus or minus one, hence $\det R=1$. Set $V=RU{R}^T$. It is symmetric positive definite, $V^2=A{A}^T$, and $VR=RU=A$.

For uniqueness, any second factorization $A=R_1{U}_1$ with the same properties gives $U_1^2=A^{T}A=C$. Since $U_1$ commutes with its square, it preserves every eigenspace of $C$. On a $c$-eigenspace its own positive eigenvalues must square to $c$, so it equals $\sqrt{c}$ times the identity there. Thus $U_1=U$, then $R_1=A{U}^{-1}=R$, and finally $V$ is fixed too. This proves both uniqueness and the polar decomposition in continuum mechanics, with the same proper rotation in its left and right forms.

Use the curvature sign convention implicit in part (c): the Jacobi field equation is $D_t^{2}J+K_{\dot{\gamma }}J=0$, so the operator in a positively curved normal direction has positive eigenvalue. With the alternative convention $R_{\mathrm{std}}(X,Y)Z={\nabla }_X{\nabla }_{Y}Z-{\nabla }_Y{\nabla }_{X}Z-{\nabla }_{[X,Y]}Z$ used for the catalog's Jacobi curvature operator, the paper's tensor is $R=-R_{\mathrm{std}}$, and its operator is $K_v(x)=R_{\mathrm{std}}(x,v)v$. This fixes the sign throughout Q1.

Let $r(A,B,C,D)=⟨R(A,B)C,D⟩$. Pair interchange for the Riemann curvature tensor givesThus $K_v$ is self-adjoint with respect to the Riemannian metric. The curvature sign does not affect this conclusion. The spectral theorem for real symmetric matrices therefore supplies the real orthonormal eigenbasis used in part (b). Also $K_{v}v=0$, so the tangent direction is a zero-eigenvalue direction.

Put $r=k/h^2$. The update isUnder $k\le h^2/(2\beta )$, all three weights are nonnegative and sum to one. With the imposed zero Dirichlet boundary conditions, this is a convex combination including any boundary values. Thus the monotone half-grid diffusion update givesDifferences of two computed solutions obey the same estimate, proving numerical stability in the discrete maximum norm, uniformly in $h,k,n$.

There is also an L2 norm proof. The diffusion matrix $L_h$ is symmetric and satisfiesSince $(v_{m+1}-v_m{)}^2\le 2(v_{m+1}^2+v_m^2)$, every eigenvalue lies in $[-4\beta /h^2,0]$. Hence the eigenvalues of $I+k{L}_h$ lie in $[-1,1]$ under the same step restriction, and the spectral theorem for real symmetric matrices gives an L2 norm contraction. These stability estimates are valid for the printed recurrence, although they cannot repair its inconsistency with the printed advection equation.

At a fixed material point, freeze the incremental elastic moduli at their local values and fix a unit propagation direction $\nu$. For an arbitrary nonzero polarization $m$, the rank-one matrix $a_{i\alpha }=m_i{\nu }_{\alpha }$ annihilates the added quadratic minor null Lagrangian, sinceThe assumed positivity therefore givesThus the acoustic tensor is positive-definite; this is strong ellipticity in elasticity. Major symmetry also makes it a real symmetric matrix. By the spectral theorem for real symmetric matrices, it has three real positive eigenvalues $q_1,q_2,q_3$ and an orthonormal basis of real polarizations $m^{(1)},m^{(2)},m^{(3)}$. ThereforeEach polarization gives a real plane wave profile through the equation of part (a), in either sign of the propagation speed. Repeated eigenvalues still admit three independent polarizations; the claim is not that the three speeds must be distinct. For an inhomogeneous base deformation these are local, frozen-coefficient wave modes, rather than exact plane waves throughout the entire body.

By the spectral theorem for real symmetric matrices, choose an orthonormal eigenbasis $u_1,u_2,u_3$ with $A{u}_i={\lambda }_i{u}_i$. If $x={\sum }_i{y}_i{u}_i$, thenThus every point of the ellipsoid has distance at least ${\lambda }_1^{-1/2}$ from the origin. Equality holds precisely whenBoth coefficients are strictly positive, so $y_2=y_3=0$. The surface equation then gives $y_1=\pm {\lambda }_1^{-1/2}$. ThereforeExactly two points attain the minimum. The strict ordering makes the largest-eigenvalue eigenspace one-dimensional; this is the equality case in nearest points to the origin on a positive-definite ellipsoid.

For the polar decomposition of an invertible real matrix, start with $C=F^{T}F$. For nonzero $y$, $y^{T}Cy=|Fy{|}^2>0$, since $\det F>0$ makes $F$ invertible. The spectral theorem for real symmetric matrices gives an orthonormal basis $n_a$ with $C{n}_a={\lambda }_a^2{n}_a$ and ${\lambda }_a>0$. DefineThen $U$ is a positive-definite symmetric matrix, $U^2=C$, andbecause $\det U=\sqrt{\det (F^{T}F)}=\det F$. Thus $R$ is an orthogonal matrix with positive determinant. Set $V=RU{R}^T$; it too is a positive-definite symmetric matrix, and $VR=RU=F$. Moreover $V^2=F{F}^T$. This provesThe positive square root is unique: any positive symmetric square root of $C$ commutes with $C$, preserves its eigenspaces and has the positive scalar square root on each of them. Hence $U$, then $R$ and $V$, are unique. These are the right stretch tensor, rotation and left stretch tensor of the polar decomposition in continuum mechanics.

A standard spectral strain measure isTake $f\in C^1(0,\infty )$, $f(1)=0$ and $f^{\prime }(1)=1$, with $f$ strictly increasing; the customary stronger condition $f^{\prime }(\lambda )>0$ gives a regular one-to-one parameterization of all positive principal stretches. The zero condition makes a rigid rotation strain free, monotonicity distinguishes extension from contraction and prevents nonunit stretch from being strain free, and the derivative normalization gives the ordinary infinitesimal strain. Indeed, for $F=I+H$,Under a superposed spatial rotation $F\mapsto QF$, $F^{T}F$ and hence $E^f$ remain unchanged, giving material frame indifference. Choosing $f(\lambda )=\lambda -1$ satisfies all these conditions and gives the Biot strain tensor

The meaning of work-conjugate stress and strain here is that a symmetric material tensor $T^f$ satisfiesfor every admissible rate, with both powers measured per reference volume. To keep the nominal-stress index convention explicit, write $M_{iI}=P_{Ii}$, so $M=P^T$ and the power is $M:\dot{F}$. Differentiating $F=RU$ gives $\dot{F}=R(\Omega U+\dot{U})$, where $\Omega =R^T\dot{R}$ is skew. Conservation of angular momentum gives symmetry of $M{F}^T$, or equivalently of $R^{T}MU$. ThereforeThe first term vanishes because a symmetric tensor has zero contraction with a skew tensor. Since ${\dot{E}}^{(1)}=\dot{U}$, the symmetric Biot stress isIf the nominal stress is instead stored spatial-index first as $M$, the same answer is $\mathrm{sym}(R^{T}M)$. The symmetrization is essential: $PR$ need not itself be symmetric.

Principal component analysis replaces the original coordinates by orthogonal linear combinations, ordered so that the first few retain as much variance as possible. It gives a lower-dimensional summary for visualization or compression and reveals the main directions of variation. A direction of large variance need not be the direction most useful for classification.

Write $\mu =\mathbb{E}X$. For a unit vector $u$, the centered score $Y=u^T(X-\mu )$ has varianceMaximizing this Rayleigh quotient with a Lagrange multiplier givesThus a maximizing direction must be an eigenvector of the covariance matrix. To identify the maximum, use the spectral theorem for real symmetric matrices to choose an orthonormal basis of eigenvectors $v_1,\dots ,v_p$, with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. Writing $u={\sum }_j{c}_j{v}_j$ givesEquality is attained at $u=v_1$. Consequently the first population principal component isFor each subsequent principal component, maximize the same variance over unit coefficient vectors orthogonal to all the previously chosen ones. Restricting the expansion above to the remaining eigenvectors gives $v_2$, then $v_3$, and so on. ThusThe principal components are uncorrelated; they need not be independent unless additional distributional assumptions, such as joint Gaussianity, hold. Their total variance is $\mathrm{tr}\Sigma ={\sum }_j{\lambda }_j$. Signs of eigenvectors are arbitrary, and a repeated eigenvalue permits any orthonormal basis in its eigenspace. In sample principal component calculations, replace $\mu$ by the sample mean and $\Sigma$ by the sample covariance matrix.

Two real symmetric bilinear forms are congruent if an invertible linear map takes one to the other; in matrices, $B=S^{\mathsf{T}}AS$ for some invertible $S$. The spectral theorem for real symmetric matrices gives an orthogonal eigenbasis. Rescaling each basis vector with nonzero eigenvalue then reduces any such form toSylvester's law of inertia asserts that $(p,q,r)$ is independent of the congruence reduction. To prove it, $p$ is the greatest dimension of a subspace on which the form is positive definite: the positive coordinate subspace attains it, whereas any subspace of dimension greater than $p$ meets the nonpositive coordinate subspace of dimension $n-p$ and therefore cannot be positive definite. Applying the same argument to the negative of the form determines $q$. Then $r=n-p-q$ is determined too.

The rank of a quadratic form is $p+q$, and its signature of a quadratic form is $p-q$ under the signed-difference convention. They determine $p,q,r$ and hence the congruence class. Counting all pairs $p,q\ge 0$ with $p+q\le n$ givesFor $n\ge 1$, exactly two classes have every level set bounded: the positive-definite and negative-definite classes. Their nonempty nonzero levels are ellipsoids, and their zero level is the origin. A degenerate form has an entire null line in its zero level. An indefinite nondegenerate form also has a nonzero isotropic vector, for example the sum of one positive and one negative normalized basis vector, whose scalar multiples make the zero level unbounded.

For the displayed three-dimensional form,Its leading principal minors are $3,14,32$, all positive. Sylvester's criterion makes it positive definite, so the surface is a bounded ellipsoid. Directly, its quadratic expression iswhich also proves positivity and boundedness of the unit level.

Use the Euclidean norm and its induced operator norm. The spectral theorem for real symmetric matrices gives $\Vert A\Vert ={\lambda }_n$ and $\Vert A^{-1}\Vert =1/{\lambda }_1$. If $e=y_{\delta }-y$, then the direct inverse has error $A^{-1}e$, soFor nonzero $y$, the relation $\Vert y\Vert =\Vert Ax\Vert \le {\lambda }_n\Vert x\Vert$ also givesThis is the spectral condition number of a positive-definite matrix. The worst relative amplification is attained by exact data along an eigenvector for ${\lambda }_n$ and noise along one for ${\lambda }_1$. A large $\kappa$ therefore makes the reconstruction sensitive to relative data errors. Absolute sensitivity additionally depends on ${\lambda }_1$ itself: scaling every eigenvalue equally leaves $\kappa$ unchanged but changes $1/{\lambda }_1$. Since the matrix is finite dimensional and positive definite, the inverse exists and is continuous; the issue is ill-conditioning rather than mathematical nonexistence or discontinuity.

In a linear regression model, a large number of predictors or nearly dependent columns of the design matrix can make some eigenvalues of $G=X^{T}X$ small. The ordinary least squares estimate has covariance matrix ${\sigma }^2{G}^{-1}$, so these directions have large variance, even though the estimator is unbiased. Ridge regression reduces this variance by shrinking the estimate towards zero, at the cost of introducing bias. This bias-variance tradeoff can improve the mean squared error when the signal is not too large in the shrunken directions.

For the usual preprocessing with a regression intercept, separate the constant column from the predictors and leave the intercept unpenalized. Subtract the sample mean from the response and from each nonconstant predictor, then divide each centered predictor column by its positive scale, for example its Euclidean norm. Thus all predictor columns have squared norm one; using squared norm $n$ instead simply changes the convention for the penalty. Centering is performed on the slope columns, not on the constant column. A full-rank design containing an intercept retains full rank on its centered slope columns. A model without an intercept should retain its specified mean structure unless an intercept is deliberately introduced.

Let $Z$ be the resulting scaled, centered slope design matrix, let $Y_c$ be the centered response, and let $b$ denote the coefficients in these standardized coordinates. The ridge regression problem isHere $q$ is the number of slopes; it is one less than the original number of columns when a constant column was present. The gradient of this objective is $2(Z^{T}Z+\lambda I)b-2Z^T{Y}_c$. Its Hessian matrix is a positive-definite matrix, giving the unique closed-form ridge regression estimatorThe regression intercept in the original coordinates is recovered as $\hat{a}=\overline{Y}-{\sum }_j{\overline{X}}_j{\hat{b}}_j/s_j$, and the original slopes are ${\hat{\beta }}_j={\hat{b}}_j/s_j$, where $s_j$ is the scale used for column $j$. This is the unpenalized intercept in ridge regression convention. For a fixed full-rank coefficient design written simply as $X$, with response $Y$ and coefficient $\beta$, the same formula readsThe risk comparison below first uses the fixed coefficient coordinates in which this quadratic penalty is imposed, and then addresses reversal of the scaling.

Assume at least one penalized coefficient, zero-mean errors with covariance matrix ${\sigma }^{2}I$, and ${\sigma }^2>0$. No normal distribution assumption is needed for this mean squared error calculation. Put $G=X^{T}X$ and $A_{\lambda }=(G+\lambda I{)}^{-1}$. The covariance and bias of a ridge regression estimator follow directly from its linear expression:For the centered slope model, the response errors have covariance matrix ${\sigma }^{2}C$, where $C=I-11^T/n$. Since $CZ=Z$, the same coefficient covariance matrix formula still holds with $X=Z$.

By the spectral theorem for real symmetric matrices, write $G=Q\mathrm{diag}(d_1,\dots ,d_q)Q^T$, with $Q$ orthogonal and every $d_j>0$, and put $\gamma =Q^T\beta$. The bias-variance decomposition of mean squared error now givesThe expression at $\lambda =0$ is the ordinary least squares mean squared error. Its right derivative isTherefore, for every fixed coefficient vector, all sufficiently small positive penalties strictly improve mean squared error.

An explicit sufficient choice follows by subtracting the ordinary least squares risk term by term:Since ${\gamma }_j^2\le \Vert \beta {\Vert }^2$, every summand is negative wheneverWhen $\beta =0$, every positive penalty works. This is a pointwise result: the permitted penalty may depend on the true coefficient vector. A fixed positive penalty cannot improve risk for all arbitrarily large signals, because its squared bias grows quadratically with the signal. The same distinction appears in uniform directional risk improvement by ridge regression and unbounded directional risk of fixed ridge shrinkage. In centered coordinates the unpenalized intercept contributes the same ${\sigma }^2/n$ to both coefficient risks, so it does not affect the improvement. Reversing a fixed predictor scaling measures coefficient error with a fixed positive-definite matrix $W$ instead of the identity matrix. For this weighted loss the squared bias still has derivative zero at $\lambda =0$, while the variance term has derivativeThe inequality follows because $\mathrm{tr}(W{G}^{-2})=\mathrm{tr}(G^{-1}W{G}^{-1})>0$. Thus sufficiently small penalties also improve the original-coordinate mean squared error, though the explicit sufficient bound above was for the unweighted coordinates. Recovering an uncentered intercept adds a fixed nonnegative quadratic form to the slope loss, and its sample-mean error is uncorrelated with the centered slope errors, so the same weighted argument applies. A model with only an unpenalized intercept gives identical estimators. If the noise variance were zero, the strict improvement assertion would fail; positive noise variance is essential.

Take $z$ off the real axis, or more generally require both $X-zI$ and $X^{(i)}-zI$ to be invertible. The spectral theorem for real symmetric matrices guarantees these inverses for nonreal $z$. Move coordinate $i$ to the first position by a simultaneous row and column permutation. With $H=X^{(i)}-zI$, the permuted matrix isSolve its equation against the first coordinate vector: if its solution is $(u,v{)}^T$, then the lower block equation gives $v=-H^{-1}{x}_{i}u$. Substitution into the first equation gives $(X_{ii}-z-x_i^T{H}^{-1}{x}_i)u=1$. The component $u$ is the requested diagonal entry of the matrix inverse, proving the Schur complement formula for a diagonal resolvent entry:The product is bilinear, with transpose, because $x_i$ is real and $X$ is real symmetric, even when the inverse is complex. In a complex Hermitian matrix version the row is $x_i^{\ast }$ instead. The final $x_i$ belongs inside the denominator, as in the original PDF; the converted TeX misplaces it outside the fraction.

The inverse hypotheses matter if $z$ is real: $X=\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)$ is invertible at $z=0$, while its one-by-one principal minor is not. Thus existence of the full inverse alone is not sufficient to use this formula.

Put $\eta =\mathrm{Im}z>0$. For a real eigenvalue $\lambda$,The spectral theorem for real symmetric matrices therefore gives $\mathrm{Im}g>0$ and $\mathrm{Im}{q}_i\ge 0$, since $q_i$ is a sum of $(\lambda -z{)}^{-1}$ with nonnegative coefficients given by squared eigenvector coordinates of $x_i$. This is the positive imaginary part of a Stieltjes matrix resolvent. HenceApply the triangle inequality to the identity above, and add and subtract the minor trace:The allowed principal minor resolvent trace bound, with the printed $N$ normalization, bounds the second average by $c/(\eta N)$. Thus the explicit conclusion isIn particular, the final term is $O((\mathrm{Im}z{)}^{-3}{N}^{-1})$ with a constant independent of $N$. The upper-half-plane restriction gives the displayed positive denominators; in the lower half-plane the analogous bound uses $|\mathrm{Im}z|$.

The chosen centering also has a probabilistic meaning. The column vector $x_i$ is independent of the minor, has mean zero and coordinate variance $1/N$, so its conditional expectation satisfiesThis explains why the bound isolates fluctuations of a quadratic form around the minor trace. Mean centering alone is not a concentration estimate; no unrequested limiting assertion is being assumed.

With the same squared-error normalization as in Question 2, the ridge estimator solvesThe unpenalized intercept in ridge regression is $\hat{a}=\overline{Y}$ because the columns of $X$ are centered. Differentiating the remaining quadratic objective gives $(X^{T}X+n\lambda I_p){\hat{\beta }}_{\lambda }^R=X^T{Y}_c$. Since $\lambda >0$, the left matrix is positive definite even when $X$ is rank-deficient. Thus the closed-form ridge regression estimator isThe second equality again uses centering. If the penalty convention is $\lambda \Vert b{\Vert }_2^2/2$ with an unnormalized squared-error term $\Vert Y_c-Xb{\Vert }_2^2/2$, replace $n\lambda$ by $\lambda$ throughout; the limiting assertion is unchanged.

For the full-column-rank case, take a singular value decomposition $X=UD{V}^T$ with positive singular values $d_1,\dots ,d_p$. The ordinary least squares estimator, which is the slope maximum-likelihood estimator in this normal linear model, and the ridge regression estimator are respectivelyConsequently principal-component shrinkage by ridge regression multiplies the least-squares coordinate in direction $v_j$ by $d_j^2/(d_j^2+n\lambda )$. Ridge shrinks most strongly in directions with the smallest singular values. These are combinations of predictors that the data distinguish least well.

Near multicollinearity creates small singular values, and the factor $1/d_j$ in ordinary least squares greatly amplifies noise in those directions. Its coefficient variance there is ${\sigma }^2/d_j^2$, while the ridge variance iswhich is smaller. The price is bias: the expectation of the ridge coordinate is $d_j^2/(d_j^2+n\lambda )$ times the corresponding true coefficient. This bias-variance tradeoff can substantially reduce estimation and prediction error when the poorly identified directions do not contain an excessively large signal. It is not a guarantee of improvement for every coefficient vector. Adding $n\lambda$ to every eigenvalue also improves the condition number of the normal matrix and makes numerical inversion more stable.

To handle every rank and every relation between $n$ and $p$, put $G=X^{T}X$ and use its spectral theorem for real symmetric matrices. Choose an orthonormal eigenbasis $v_1,\dots ,v_p$ with eigenvalues ${\gamma }_j\ge 0$, and put $z=X^{T}Y$. ThenIf ${\gamma }_j=0$, then $\Vert X{v}_j{\Vert }_2^2=v_j^{T}G{v}_j=0$, so $X{v}_j=0$ and $v_j^{T}z=(X{v}_j{)}^{T}Y=0$. Thus these terms are exactly zero for every positive $\lambda$; there is no divergent component in the null space. Each remaining term converges to $(v_j^{T}z/{\gamma }_j)v_j$. By the definition of the Moore-Penrose pseudoinverse,and thereforeThis proves the vanishing-penalty ridge limit without an invertibility assumption. The limit is the minimum-norm least-squares solution: every other least-squares solution differs by a vector in $\ker X$, orthogonal to the displayed solution, and therefore has at least as large a Euclidean norm.