= Solution
<Principal component analysis> replaces correlated measurements by orthogonal linear coordinates ordered by their <variance>. For centered observations collected in an $n\times p$ <matrix> $Z$, write the <sample covariance matrix> as $S=Z^TZ/(n-1)$. A unit loading vector $a$ gives score vector $Za$ with sample <variance> $a^TSa$. The <Rayleigh quotient> is maximized by a unit <eigenvector> of $S$ for its largest <eigenvalue>.
The subsequent <principal components> maximize the same quadratic form subject to orthogonality to the earlier loading vectors. By the <spectral theorem>, if $S=Q\Lambda Q^T$ with descending eigenvalues, the scores $ZQ$ have diagonal covariance $\Lambda$. Their fractions of explained <variance> are $\lambda_j/\sum_i\lambda_i$. Repeated <eigenvalues> identify an eigenspace rather than a unique axis.
Keeping the first $q$ <principal components> gives the rank-$q$ reconstruction $\widehat Z=ZQ_qQ_q^T$. Its squared reconstruction error is
$$
\|Z-\widehat Z\|_F^2=(n-1)\sum_{j>q}\lambda_j.
$$
For any rank-$q$ orthogonal projection $P$, retained variance is $\operatorname{tr}(SP)=\sum_j\lambda_j q_j^TPq_j$. The weights $q_j^TPq_j$ lie in $[0,1]$ and sum to $q$, so this is at most the sum of the largest $q$ eigenvalues. Projecting onto any candidate reconstruction subspace is its best least-squares reconstruction; this proves the minimum-error property and agrees with the <singular value decomposition>. Thus \b[PCA finds the linear subspace preserving the greatest variance, equivalently minimizing squared reconstruction error.] The upper-left sketch shows the first axis aligned with the elongated cloud; the second is perpendicular. A <scree plot> and substantive interpretability help choose $q$.
\Image[/past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2001/iii/paper-30-multivariate-sketches.png]
{title=Original sketches of principal components, classical scaling, hierarchical clustering and multivariate mean comparison}
These methods depend on measurement scale. A variable measured in large numerical units can dominate covariance-based <principal component analysis>; standardizing variables gives <principal component analysis on a correlation matrix>. Scores summarize patterns, but the method neither establishes causation nor guarantees that the largest-variance directions best predict a separate response.
Back to article page
