= Solution
<Principal component analysis> replaces correlated measurements by uncorrelated linear compounds ordered by decreasing <variance>. Keeping the leading compounds gives a low-dimensional summary that preserves as much <variance> as possible under orthonormal projection. It is an unsupervised description of variation, rather than a guarantee that the high-variance directions are best for classification or prediction.
Write $X_c=X-\mu$. For a unit loading vector $q$, the <variance> of $q^TX_c$ is $q^T\Sigma q$. The <spectral theorem> gives an orthonormal eigenbasis $q_1,\ldots,q_p$ with <eigenvalues> $\lambda_1\ge\cdots\ge\lambda_p\ge0$. If $q=\sum_jb_jq_j$ and $\sum_jb_j^2=1$, then
$$
q^T\Sigma q=\sum_j\lambda_jb_j^2\le\lambda_1.
$$
Equality is attained at $q_1$. Requiring the next loading to be orthogonal to the first excludes its eigendirection, so the same argument gives $q_2$ and <variance> $\lambda_2$, and continues inductively. Repeated <eigenvalues> permit any <orthonormal basis> of the tied eigenspace. These are the <population principal components>.
With $Q=(q_1,\ldots,q_p)$, set $Y=Q^TX_c$. Then
$$
\boxed{\operatorname{Cov}(Y)=\Lambda=Q^T\Sigma Q
=\operatorname{diag}(\lambda_1,\ldots,\lambda_p),\qquad
\Sigma=Q\Lambda Q^T.}
$$
In particular the components are uncorrelated. They need not be independent unless, for example, $X$ is <multivariate normal>. Finally
$$
\boxed{\sum_{j=1}^p\operatorname{Var}(Y_j)
=\sum_{j=1}^p\lambda_j
=\operatorname{tr}(\Sigma)
=\sum_{j=1}^p\operatorname{Var}(X_j).}
$$
The <trace> equality follows from $Q^TQ=I$ and cyclicity of <trace>, so the rotation preserves total <variance>. The <explained variance of a principal component> is its <eigenvalue> divided by this total.
Back to article page
