Past exam of the mathematics course of the University of Cambridge / 2006 / iii / Paper 46 / 3 / a / Solution

Principal component analysis replaces the original coordinates by orthogonal linear combinations, ordered so that the first few retain as much variance as possible. It gives a lower-dimensional summary for visualization or compression and reveals the main directions of variation. A direction of large variance need not be the direction most useful for classification.

Write $\mu =\mathbb{E}X$. For a unit vector $u$, the centered score $Y=u^T(X-\mu )$ has varianceMaximizing this Rayleigh quotient with a Lagrange multiplier givesThus a maximizing direction must be an eigenvector of the covariance matrix. To identify the maximum, use the spectral theorem for real symmetric matrices to choose an orthonormal basis of eigenvectors $v_1,\dots ,v_p$, with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. Writing $u={\sum }_j{c}_j{v}_j$ givesEquality is attained at $u=v_1$. Consequently the first population principal component isFor each subsequent principal component, maximize the same variance over unit coefficient vectors orthogonal to all the previously chosen ones. Restricting the expansion above to the remaining eigenvectors gives $v_2$, then $v_3$, and so on. ThusThe principal components are uncorrelated; they need not be independent unless additional distributional assumptions, such as joint Gaussianity, hold. Their total variance is $\mathrm{tr}\Sigma ={\sum }_j{\lambda }_j$. Signs of eigenvectors are arbitrary, and a repeated eigenvalue permits any orthonormal basis in its eigenspace. In sample principal component calculations, replace $\mu$ by the sample mean and $\Sigma$ by the sample covariance matrix.