= Solution
For fixed $A$, differentiating under the constraint $\sum_i z_i=0$ gives
$$
\widehat\mu=\overline X,
\qquad
\widehat z_i=(A^TA)^{-1}A^T(X_i-\overline X).
$$
Because the columns $u_j$ are orthogonal,
$$
\widehat z_{ij}
=\frac{u_j^T(X_i-\overline X)}{\|u_j\|_2^2}.
$$
The fitted value is therefore $\overline X+\Pi_A(X_i-\overline X)$, where $\Pi_A$ is the <orthogonal projection> onto the column space of $A$.
The residual sum of squares is minimized by choosing this column space to be the span of the $d$ leading eigenvectors of
$$
\sum_i(X_i-\overline X)(X_i-\overline X)^T.
$$
Thus $\widehat A$ may be taken to have those orthonormal eigenvectors as columns. Their nonzero scales are immaterial because inverse scaling of the scores leaves $Az_i$ unchanged.
Back to article page
