 Solution

Principal component analysis replaces correlated measurements by uncorrelated linear compounds ordered by decreasing variance. Keeping the leading compounds gives a low-dimensional summary that preserves as much variance as possible under orthonormal projection. It is an unsupervised description of variation, rather than a guarantee that the high-variance directions are best for classification or prediction.

Write $X_c=X-\mu$. For a unit loading vector $q$, the variance of $q^T{X}_c$ is $q^T\Sigma q$. The spectral theorem gives an orthonormal eigenbasis $q_1,\dots ,q_p$ with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. If $q={\sum }_j{b}_j{q}_j$ and ${\sum }_j{b}_j^2=1$, thenEquality is attained at $q_1$. Requiring the next loading to be orthogonal to the first excludes its eigendirection, so the same argument gives $q_2$ and variance ${\lambda }_2$, and continues inductively. Repeated eigenvalues permit any orthonormal basis of the tied eigenspace. These are the population principal components.

With $Q=(q_1,\dots ,q_p)$, set $Y=Q^T{X}_c$. ThenIn particular the components are uncorrelated. They need not be independent unless, for example, $X$ is multivariate normal. FinallyThe trace equality follows from $Q^{T}Q=I$ and cyclicity of trace, so the rotation preserves total variance. The explained variance of a principal component is its eigenvalue divided by this total.