Past exam of the mathematics course of the University of Cambridge / 2022 / iii / Paper 218 / 3 / e / Solution

For fixed $A$, differentiating under the constraint ${\sum }_i{z}_i=0$ givesBecause the columns $u_j$ are orthogonal,The fitted value is therefore $\overline{X}+{\Pi }_A(X_i-\overline{X})$, where ${\Pi }_A$ is the orthogonal projection onto the column space of $A$.

The residual sum of squares is minimized by choosing this column space to be the span of the $d$ leading eigenvectors ofThus $\hat{A}$ may be taken to have those orthonormal eigenvectors as columns. Their nonzero scales are immaterial because inverse scaling of the scores leaves $A{z}_i$ unchanged.