Past exam of the mathematics course of the University of Cambridge / 2019 / ii / Paper 1 / 29J (Principles of Statistics)

The Gaussian likelihood is proportional toso maximum likelihood estimation is equivalent to ordinary least squares. If $p\le n$ and $X$ has rank $p$, then $X^{T}X$ is invertible and the normal equations giveIf $p>n$, then $\mathrm{rank}X<p$ and $\ker X\ne \{0\}$. A least-squares minimizer always exists, but it is not unique: by rank-deficient ordinary least squares, the complete set isan affine subspace of dimension $p-\mathrm{rank}X>0$. Thus there are uncountably infinitely many likelihood maximizers. In the generic full-row-rank case their dimension is $p-n$.

Differentiating the ridge regression objective givesFor every nonzero $z$,so $X^{T}X+\lambda I_p$ is positive definite and hence invertible. The strictly convex objective therefore has the unique closed-form ridge regression estimator

Because every column of $X$ has sample mean zero, every column $u_i$ of $U=XV$ also has sample mean zero. The spectral theorem for real symmetric matrices givesHence for $i\ne j$ the sample covariance is $n^{-1}{u}_i^T{u}_j=0$, while the sample variance of $u_i$ isThus the $u_i$ are pairwise uncorrelated sample principal components, ordered by decreasing sample variance.

Since $X=U{V}^T$ and $X^{T}X=V\Lambda V^T$,ThereforePut $w_i=u_i/\sqrt{{\Lambda }_{ii}}$. By part (c), the $w_i$ are normalized sample principal components forming an orthonormal set, andThis is the orthogonal projection onto a finite-dimensional subspace spanned by the $w_i$, so ${\hat{Y}}_{\mathrm{MLE}}$ is the unique closest point in that subspace to $Y$.

Using the estimator from part (b) and the spectral decomposition of $X^{T}X$,ThusUnder the stated separation of scales, the shrinkage factor is approximately one for $i\le q$ and approximately zero for $i>q$. The principal-component shrinkage by ridge regression formula therefore reduces approximately tothe orthogonal projection of $Y$ onto the span of the $q$ normalized sample principal components with greatest variance.