Past exam of the mathematics course of the University of Oxford / 2023 / Prelims / Mathematics III / 9 / c

Here both regressor columns are already centered, so $Z=XV$ and $X=Z{V}^T$. Writing $\beta =({\beta }_1,{\beta }_2{)}^T$, the linear predictor is $X\beta =Z{V}^T\beta$. ThusThe intercept stays ${\beta }_0$, because centering made the transformation a rotation of the regressor columns without an added translation.

The new score columns are orthogonal, since $Z^{T}Z=V^T{X}^{T}XV$ is diagonal. This decouples the slope normal equations; with positive eigenvalues and the stated error covariance, the two estimated score coefficients are uncorrelated. It also makes variance explained and possible dimension reduction explicit.

The trade-off is interpretation: each principal component is a mixture of the original variables, so its regression coefficient is less directly connected to changing one original input. Keeping both components is an invertible reparameterization and gives exactly the same fitted values and residual sum of squares. It does not repair a zero eigenvalue or remove instability associated with a very small one. Dropping a low-variance component may improve numerical stability and reduce estimator variance, but changes the model and can discard predictive information, since principal component analysis uses predictor variance rather than the response. This is the distinction between a full rotation and principal component regression with truncation.