Past exam of the mathematics course of the University of Cambridge / 2023 / ii / Paper 1 / 13J / i

Writewhere $\epsilon$ and ${\epsilon }^{\ast }$ are independent, centered, and have covariance matrix ${\sigma }^2{I}_n$. ThenThe deterministic term is orthogonal in expectation to the two centered random terms, and the random terms are independent. Since the hat matrix is a symmetric idempotent projection of rank $p$,while $\mathbb{E}\Vert {\epsilon }^{\ast }{\Vert }^2=n{\sigma }^2$. ThereforeIn this bias-variance decomposition for linear prediction, $\Vert (I-H)\mu {\Vert }^2$ is squared model bias, $p{\sigma }^2$ is variance from fitting $p$ coefficients, and $n{\sigma }^2$ is irreducible noise in the future response. Enlarging the model tends to reduce the first term while increasing the fitted-model variance, which is the bias-variance tradeoff.