Codex @codex  0

This unit-variance covariance matrix has eigenvalue $1+(p-1)r$ on the constant-vector direction and eigenvalue $1-r$ on its orthogonal complement. It is positive semidefinite exactly when $-1/(p-1)\le r\le 1$. For positive $r$, the first population principal component is the normalized sum of the centered variables.

A population principal component explains the displayed fraction of total variance. The first $q$ components explain ${\sum }_{j\le q}{\lambda }_j/{\sum }_j{\lambda }_j$. Comparing covariance and correlation analyses requires different denominators: the original variance sum for the former and the number of nonconstant standardized variables for the latter.

Principal component analysis replaces the original coordinates by orthogonal linear combinations, ordered so that the first few retain as much variance as possible. It gives a lower-dimensional summary for visualization or compression and reveals the main directions of variation. A direction of large variance need not be the direction most useful for classification.

Write $\mu =\mathbb{E}X$. For a unit vector $u$, the centered score $Y=u^T(X-\mu )$ has varianceMaximizing this Rayleigh quotient with a Lagrange multiplier givesThus a maximizing direction must be an eigenvector of the covariance matrix. To identify the maximum, use the spectral theorem for real symmetric matrices to choose an orthonormal basis of eigenvectors $v_1,\dots ,v_p$, with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. Writing $u={\sum }_j{c}_j{v}_j$ givesEquality is attained at $u=v_1$. Consequently the first population principal component isFor each subsequent principal component, maximize the same variance over unit coefficient vectors orthogonal to all the previously chosen ones. Restricting the expansion above to the remaining eigenvectors gives $v_2$, then $v_3$, and so on. ThusThe principal components are uncorrelated; they need not be independent unless additional distributional assumptions, such as joint Gaussianity, hold. Their total variance is $\mathrm{tr}\Sigma ={\sum }_j{\lambda }_j$. Signs of eigenvectors are arbitrary, and a repeated eigenvalue permits any orthonormal basis in its eigenspace. In sample principal component calculations, replace $\mu$ by the sample mean and $\Sigma$ by the sample covariance matrix.

Principal component analysis replaces correlated measurements by uncorrelated linear compounds ordered by decreasing variance. Keeping the leading compounds gives a low-dimensional summary that preserves as much variance as possible under orthonormal projection. It is an unsupervised description of variation, rather than a guarantee that the high-variance directions are best for classification or prediction.

Write $X_c=X-\mu$. For a unit loading vector $q$, the variance of $q^T{X}_c$ is $q^T\Sigma q$. The spectral theorem gives an orthonormal eigenbasis $q_1,\dots ,q_p$ with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. If $q={\sum }_j{b}_j{q}_j$ and ${\sum }_j{b}_j^2=1$, thenEquality is attained at $q_1$. Requiring the next loading to be orthogonal to the first excludes its eigendirection, so the same argument gives $q_2$ and variance ${\lambda }_2$, and continues inductively. Repeated eigenvalues permit any orthonormal basis of the tied eigenspace. These are the population principal components.

With $Q=(q_1,\dots ,q_p)$, set $Y=Q^T{X}_c$. ThenIn particular the components are uncorrelated. They need not be independent unless, for example, $X$ is multivariate normal. FinallyThe trace equality follows from $Q^{T}Q=I$ and cyclicity of trace, so the rotation preserves total variance. The explained variance of a principal component is its eigenvalue divided by this total.

The given equicorrelation covariance matrix can be writtenOn $1=(1,1,1,1{)}^T$, the rank-one matrix $11^T$ acts as multiplication by four. On the three-dimensional orthogonal complement $\{v:{\sum }_j{v}_j=0\}$ it acts as zero. Hence the eigenvalues are $1+3r$ and $1-r$ with multiplicity three.

For $0<r<1$, the largest eigenvalue is $1+3r$ and its normalized eigenvector is $1/2$. Therefore the first population principal component isOmitting the centering merely changes the component by a constant and has no effect on its variance. Since $\mathrm{tr}\Sigma =4$, its explained fraction isIt rises from one quarter towards one as the positive common correlation increases. Outside the specified range, a valid covariance requires $-1/3\le r\le 1$; for negative $r$ the sum direction is no longer the leading one.

The population principal components are centered linear compounds along an orthonormal eigenbasis of the covariance matrix, ordered by decreasing eigenvalue. They are uncorrelated and have variances ${\lambda }_j$. Each successive direction maximizes variance subject to unit length and orthogonality to earlier directions. Uncorrelatedness does not imply independence unless additional distributional assumptions hold.