Principal component analysis

Principal component analysis finds orthogonal directions of greatest sample variance. Its first loading vector is a unit eigenvector corresponding to the largest eigenvalue of the sample covariance matrix.

Regress a response on selected principal component scores of centered predictors. If $X=UD{V}^T$ is a singular value decomposition and the first $q$ positive-singular-value components are retained, the retained score matrix is $U_q{D}_q$ and the fitted coefficient in predictor coordinates is $V_q{D}_q^{-1}{U}_q^{T}y$. The fitted centered response is $U_q{U}_q^{T}y$: this follows by applying the least-squares normal equations to the orthogonal score columns. Truncating small singular directions limits variance amplification, but directions of large predictor variance need not be those best predicting the response; select $q$ using prediction assessment rather than explained predictor variance alone.

A scree plot graphs ordered covariance or correlation eigenvalues against component number. A transition from a steep decline to a flatter tail suggests a dimension, but this is a diagnostic rather than a proof of the number of latent factors. In factor analysis, it should be combined with model fit, substantive interpretation and uncertainty assessment.

Two conventions occur for principal-component loadings: the unit eigenvector ${\ell }_j$ defining a principal component, and the scaled vector $r_j=\sqrt{{\lambda }_j}{\ell }_j$. For principal component analysis on a correlation matrix, the entries of the latter are the correlations between each standardized variable and the component scaled to variance one. Indeed, $\mathrm{Cov}(Z,{\ell }_j^{T}Z)=C{\ell }_j={\lambda }_j{\ell }_j$ and $\mathrm{Var}({\ell }_j^{T}Z)={\lambda }_j$. Thus $\Vert r_j{\Vert }^2={\lambda }_j$, whereas $\Vert {\ell }_j\Vert =1$; reporting the convention avoids confusing component directions with correlations.

A principal component is the projection of centered data onto a unit eigenvector of its covariance matrix. Ordering the components by their eigenvalues orders their variances. Repeated eigenvalues allow any orthonormal basis of the corresponding eigenspace.

A score is an observation's coordinate along a principal component direction. Center an observation by the fitted sample mean, apply the fitted variable scaling when using principal component analysis on a correlation matrix, then take its inner product with the unit principal component loading vector. New observations must use the training center and scaling. The score variance equals the corresponding eigenvalue when the covariance normalization is consistent.

A population principal component explains the displayed fraction of total variance. The first $q$ components explain ${\sum }_{j\le q}{\lambda }_j/{\sum }_j{\lambda }_j$. Comparing covariance and correlation analyses requires different denominators: the original variance sum for the former and the number of nonconstant standardized variables for the latter.

For positive marginal variances, standardizing each variable to variance one converts covariance-based PCA on $Z$ into correlation-based PCA on $X$. This makes the analysis insensitive to the original measurement units but changes the optimization criterion. Standardization gives small-variance variables the same marginal weight as large-variance variables; it is a modelling choice, not an automatic improvement.

The population principal components are centered linear compounds along an orthonormal eigenbasis of the covariance matrix, ordered by decreasing eigenvalue. They are uncorrelated and have variances ${\lambda }_j$. Each successive direction maximizes variance subject to unit length and orthogonality to earlier directions. Uncorrelatedness does not imply independence unless additional distributional assumptions hold.

Let the columns of a centered design matrix $X\in {\mathbb{R}}^{n\times p}$ be variables and write $X^{T}X=V\Lambda V^T$ by the spectral theorem for real symmetric matrices. The columns $u_i$ of $U=XV$ are the sample principal-component score vectors. They satisfy $u_i^T{u}_j={\Lambda }_{ij}$, so distinct score vectors have zero sample covariance and $u_i$ has sample variance ${\Lambda }_{ii}/n$ under the divisor-$n$ convention.

When ${\Lambda }_{ii}>0$, the normalized sample principal component $w_i=u_i/\sqrt{{\Lambda }_{ii}}$ has Euclidean norm one. The vectors $w_i$ form an orthonormal set spanning the column space of a full-column-rank centered design matrix.