Past exam of the mathematics course of the University of Cambridge / 2021 / ii / Paper 4 / 13J (Statistical Modelling)

Differentiating the ridge objective gives $(X^{T}X+\lambda I)\hat{\beta }=X^{T}Y$, hence $\hat{\mu }=X(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y=H_{\lambda }Y$.

Since $Y-HY=(I-H)\mu +(I-H)\epsilon$, the cross term has mean zero and $H$ is symmetric. Thus the expectation is $\Vert (I-H)\mu {\Vert }^2+{\sigma }^2\mathrm{tr}(I-H{)}^2$, which expands to the stated expression.

Independence gives $\mathbb{E}\Vert Y^{\ast }-HY{\Vert }^2=\Vert (I-H)\mu {\Vert }^2+n{\sigma }^2+{\sigma }^2\mathrm{tr}{H}^2$. Comparing with part (b) shows the difference is $2{\sigma }^2\mathrm{tr}H$, proving unbiasedness.

Here $H=hI$, $h=(1+\lambda {)}^{-1}$, and prediction risk isIt decreases then increases, with $h^{\ast }=\Vert \mu {\Vert }^2/(\Vert \mu {\Vert }^2+n{\sigma }^2)$, equivalently ${\lambda }^{\ast }=n{\sigma }^2/\Vert \mu {\Vert }^2$. Its minimum is $n{\sigma }^2+n{\sigma }^2\Vert \mu {\Vert }^2/(\Vert \mu {\Vert }^2+n{\sigma }^2)$; the endpoint limits are $2n{\sigma }^2$ and $n{\sigma }^2+\Vert \mu {\Vert }^2$.