= Solution
Put $f=(f(x_1),\ldots,f(x_n))^T$, so $Y=f+\varepsilon$ and $Y^*=f+\varepsilon^*$ with independent noise vectors having <covariance matrix> $I_n$. For any deterministic <linear smoother> $H$,
$$
\mathbb E\|Y-HY\|_2^2
=\|(I-H)f\|_2^2
+\operatorname{tr}\!\left((I-H)^T(I-H)\right),
$$
whereas independence gives
$$
\mathbb E\|Y^*-HY\|_2^2
=\|(I-H)f\|_2^2+n+\operatorname{tr}(H^TH).
$$
Expanding the first trace shows that the second expression exceeds the first by $2\operatorname{tr}(H)$, proving the identity.
For ridge regression,
$$
H_\lambda=\frac1n\mathbf1\mathbf1^T
+X(X^TX+\lambda I_p)^{-1}X^T.
$$
Its <effective degrees of freedom> are $\operatorname{tr}(H_\lambda)$. Thus training error is optimistically biased for independent-copy prediction error by $2\operatorname{tr}(H_\lambda)/n$. The graph exhibits exactly this effect: the dashed training curve keeps falling as $\lambda$ decreases, while the solid test curve eventually rises through <overfitting>.
Back to article page
