Past exam of the mathematics course of the University of Cambridge / 2006 / iii / Paper 46 / 3 / b

For $v_1=(1,1,1,0{)}^T$, direct multiplication by the sample covariance matrix givesIt is therefore an eigenvector, with eigenvalue $2250$. Normalization gives $u_1=v_1/\sqrt{3}$, so the corresponding sample principal component isIts coefficients are proportional to the required vector. The remaining eigenvalues calculated in (ii) are smaller, confirming that this is the first principal component.

Let $v_2,v_3,v_4$ be the three supplied coefficient vectors, in their displayed order. Multiplication by the sample covariance matrix givesThe four vectors are pairwise orthogonal, with norms $\sqrt{3}$, $\sqrt{102}$, $\sqrt{6}$ and $\sqrt{204}$. Hence $u_j=v_j/\Vert v_j\Vert$ form an orthonormal basis of eigenvectors. The variance of the principal component with unit coefficient vector $u_j$ is its eigenvalue ${\lambda }_j$. Using unnormalized coefficient vectors would instead give variances ${\lambda }_j\Vert v_j{\Vert }^2$ and would not produce the desired proportions.

The total variance isTherefore the explained variance of a principal component isThe first two principal components together account for $2658/3162\simeq 84.06\%$, and the first three account for $2958/3162\simeq 93.55\%$ of the total variance.

The first principal component has equal positive coefficients on the three academic subjects and zero coefficient on physical education. It measures overall academic performance in those subjects and explains $71.16\%$ of the variance. The large positive academic covariances explain why these subjects vary predominantly together.

The second principal component has normalized coefficients approximately $(0.099,0,-0.099,0.990)$. It is mainly a physical-education score, with a small adjustment for the physics-versus-mathematics contrast, and explains $12.90\%$ of the variance. The small physical-education covariances with the academic subjects make this direction close to a separate physical-education axis. This is an interpretation of the observed covariance matrix, not proof of distinct causal abilities or statistical independence.

The third principal component compares physics and mathematics together against biology: its coefficient vector is $(1,-2,1,0)/\sqrt{6}$. It captures subject balance rather than overall academic performance, accounting for $9.49\%$ of the variance. The fourth principal component primarily contrasts physics with mathematics, with a smaller physical-education correction; its normalized coefficients are approximately $(0.700,0,-0.700,-0.140)$, and it accounts for the remaining $6.45\%$. Changing all signs in any component reverses its score without changing the interpretation of its contrast or its explained variance of a principal component.

A two-component summary separates broad academic performance from predominantly physical-education performance and retains about $84.06\%$ of the variation. It still discards about $15.94\%$, including differences between academic subjects. Whether that loss is acceptable depends on the purpose: an overall summary may use two principal components, whereas studying subject-specific strengths requires the later principal components too. A single principal component would lose most of the physical-education information.

If the change is purely a rescaling of the physics mark, write $X^{\prime }=DX$, where $D=\mathrm{diag}(10/3,1,1,1)$. The sample covariance matrix transforms asThe physics variance is multiplied by $100/9$, becoming approximately $10044.44$. Each covariance between physics and another subject is multiplied by $10/3$, and all other entries stay unchanged. Principal component analysis on this new sample covariance matrix maximizes variance in the new numerical units, so its eigenvectors and explained variance of a principal component generally change. Physics receives disproportionately greater influence simply because its unit has changed.

One solution is to convert the physics marks back to the original scale before doing principal component analysis. Alternatively, use principal component analysis on a correlation matrix: standardize every subject byA positive change of units multiplies both a centered variable and its standard deviation by the same factor. Thus it leaves $Z_j$ and the sample correlation matrix $R$ unchanged.

Correlation-based principal component analysis is invariant under positive changes of units; covariance-based principal component analysis is not. Standardizing gives each subject unit variance, which is a different weighting choice and can emphasize relatively low-variance subjects. If the revised assessment changes the scores themselves rather than just their units, its sample correlation matrix may also change, and mere standardization cannot recover the original analysis.